Jul 13 14:18:51.363347 ip-10-0-131-188 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 13 14:18:51.363359 ip-10-0-131-188 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 13 14:18:51.363366 ip-10-0-131-188 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 13 14:18:51.363570 ip-10-0-131-188 systemd[1]: Failed to start Kubernetes Kubelet. Jul 13 14:19:01.485458 ip-10-0-131-188 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 13 14:19:01.485475 ip-10-0-131-188 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ff38047e0c3046bead6221da91a2d3ff -- Jul 13 14:20:58.199130 ip-10-0-131-188 systemd[1]: Starting Kubernetes Kubelet... Jul 13 14:20:58.628589 ip-10-0-131-188 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 14:20:58.628589 ip-10-0-131-188 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 13 14:20:58.628589 ip-10-0-131-188 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 14:20:58.628589 ip-10-0-131-188 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 13 14:20:58.628589 ip-10-0-131-188 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 14:20:58.631082 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.630994 2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 13 14:20:58.639216 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639195 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:20:58.639216 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639212 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:20:58.639216 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639218 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639222 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639225 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639229 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639232 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639235 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639239 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639242 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639245 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639248 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639251 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639254 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639257 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639260 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639263 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639267 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639270 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639273 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639276 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639278 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:20:58.639309 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639282 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639285 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639288 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639291 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639294 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639296 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639301 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639305 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639308 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639311 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639314 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639317 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639320 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639324 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639327 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639330 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639332 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639335 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639338 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639341 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:20:58.639810 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639343 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639346 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639349 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639351 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639354 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639357 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639360 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639362 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639365 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639368 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639371 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639373 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639376 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639379 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639383 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639386 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639389 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639391 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639394 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639397 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:20:58.640426 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639400 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639402 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639405 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639408 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639411 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639413 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639416 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639419 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639422 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639425 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639428 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639430 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639433 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639438 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639442 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639445 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639448 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639451 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639454 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:20:58.640985 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639457 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639460 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639463 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639465 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639468 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639471 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639474 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639479 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639482 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639485 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639488 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639491 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639494 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639496 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639499 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639502 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639504 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.639507 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640261 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:20:58.641490 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640268 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640271 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640275 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640278 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640294 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640297 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640300 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640303 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640305 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640308 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640313 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640317 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640320 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640323 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640326 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640329 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640332 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640335 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640338 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:20:58.641983 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640340 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640344 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640346 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640349 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640352 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640355 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640358 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640360 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640363 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640366 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640368 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640371 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640374 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640376 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640379 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640382 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640385 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640387 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640391 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:20:58.642525 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640393 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640396 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640399 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640401 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640404 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640407 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640410 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640413 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640416 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640419 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640422 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640424 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640428 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640431 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640434 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640436 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640439 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640442 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640445 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640448 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640450 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:20:58.643073 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640453 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640456 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640459 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640461 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640465 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640468 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640471 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640473 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640476 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640478 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640481 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640484 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640487 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640489 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640492 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640495 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640500 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640503 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640506 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640509 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:20:58.643635 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640512 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640515 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640518 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640522 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640525 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640528 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640531 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640534 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640537 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640540 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640543 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640545 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640550 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640553 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640557 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640560 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640563 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640566 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.640569 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:20:58.644181 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641007 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641013 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641016 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641020 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641024 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641028 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641031 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641034 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641037 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641041 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641044 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641047 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641050 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641053 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641056 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641058 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641061 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641064 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641067 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:20:58.644679 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641069 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641074 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641078 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641081 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641084 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641087 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641090 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641093 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641096 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641099 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641103 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641105 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641108 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641111 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641114 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641116 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641119 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641122 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641125 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641128 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:20:58.645217 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641130 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641134 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641140 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641142 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641145 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641148 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641151 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641153 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641156 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641158 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641161 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641164 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641166 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641169 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641172 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641175 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641177 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641180 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641183 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:20:58.645737 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641185 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641188 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641191 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641194 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641196 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641199 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641202 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641205 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641208 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641210 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641213 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641216 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641219 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641221 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641224 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641228 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641231 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641233 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:20:58.646243 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641238 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641241 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641244 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641247 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641250 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641253 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641256 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641259 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641261 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641265 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641268 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641270 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641273 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641276 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641278 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641281 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641284 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641287 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641290 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641292 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:20:58.646750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641295 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641298 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.641301 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641376 2566 flags.go:64] FLAG: --address="0.0.0.0" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641387 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641394 2566 flags.go:64] FLAG: --anonymous-auth="true" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641399 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641405 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641408 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641414 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641418 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641422 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641426 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641429 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641433 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641436 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641440 2566 flags.go:64] FLAG: --cgroup-root="" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641442 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641445 2566 flags.go:64] FLAG: --client-ca-file="" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641448 2566 flags.go:64] FLAG: --cloud-provider="external" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641451 2566 flags.go:64] FLAG: --cluster-dns="[]" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641455 2566 flags.go:64] FLAG: --cluster-domain="" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641458 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641462 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 13 14:20:58.647291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641465 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641469 2566 flags.go:64] FLAG: --container-log-max-files="5" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641472 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641475 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641479 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641482 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641485 2566 flags.go:64] FLAG: --contention-profiling="false" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641488 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641492 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641495 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641498 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641502 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641505 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641509 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641512 2566 flags.go:64] FLAG: --enable-load-reader="false" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641515 2566 flags.go:64] FLAG: --enable-server="true" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641519 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641524 2566 flags.go:64] FLAG: --event-burst="100" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641527 2566 flags.go:64] FLAG: --event-qps="50" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641531 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641534 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641537 2566 flags.go:64] FLAG: --eviction-hard="" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641541 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641544 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641547 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 13 14:20:58.647922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641550 2566 flags.go:64] FLAG: --eviction-soft="" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641553 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641556 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641559 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641562 2566 flags.go:64] FLAG: --experimental-mounter-path="" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641565 2566 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641568 2566 flags.go:64] FLAG: --fail-swap-on="true" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641571 2566 flags.go:64] FLAG: --feature-gates="" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641575 2566 flags.go:64] FLAG: --file-check-frequency="20s" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641578 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641581 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641585 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641588 2566 flags.go:64] FLAG: --healthz-port="10248" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641591 2566 flags.go:64] FLAG: --help="false" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641594 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-131-188.ec2.internal" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641597 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641600 2566 flags.go:64] FLAG: --http-check-frequency="20s" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641603 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641607 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641611 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641614 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641617 2566 flags.go:64] FLAG: --image-service-endpoint="" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641619 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 13 14:20:58.648571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641623 2566 flags.go:64] FLAG: --kube-api-burst="100" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641626 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641630 2566 flags.go:64] FLAG: --kube-api-qps="50" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641633 2566 flags.go:64] FLAG: --kube-reserved="" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641636 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641639 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641642 2566 flags.go:64] FLAG: --kubelet-cgroups="" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641645 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641648 2566 flags.go:64] FLAG: --lock-file="" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641651 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641654 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641657 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641663 2566 flags.go:64] FLAG: --log-json-split-stream="false" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641667 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641670 2566 flags.go:64] FLAG: --log-text-split-stream="false" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641672 2566 flags.go:64] FLAG: --logging-format="text" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641676 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641680 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641683 2566 flags.go:64] FLAG: --manifest-url="" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641686 2566 flags.go:64] FLAG: --manifest-url-header="" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641690 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641693 2566 flags.go:64] FLAG: --max-open-files="1000000" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641698 2566 flags.go:64] FLAG: --max-pods="110" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641701 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641706 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 13 14:20:58.649264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641709 2566 flags.go:64] FLAG: --memory-manager-policy="None" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641713 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641716 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641719 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641722 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641729 2566 flags.go:64] FLAG: --node-status-max-images="50" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641732 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641736 2566 flags.go:64] FLAG: --oom-score-adj="-999" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641739 2566 flags.go:64] FLAG: --pod-cidr="" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641744 2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641749 2566 flags.go:64] FLAG: --pod-manifest-path="" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641752 2566 flags.go:64] FLAG: --pod-max-pids="-1" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641756 2566 flags.go:64] FLAG: --pods-per-core="0" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641759 2566 flags.go:64] FLAG: --port="10250" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641763 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641766 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-092aded8b072a4b8f" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641769 2566 flags.go:64] FLAG: --qos-reserved="" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641772 2566 flags.go:64] FLAG: --read-only-port="10255" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641776 2566 flags.go:64] FLAG: --register-node="true" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641779 2566 flags.go:64] FLAG: --register-with-taints="" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641782 2566 flags.go:64] FLAG: --registry-burst="10" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641785 2566 flags.go:64] FLAG: --registry-qps="5" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641788 2566 flags.go:64] FLAG: --reserved-cpus="" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641792 2566 flags.go:64] FLAG: --reserved-memory="" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641795 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 13 14:20:58.649944 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641798 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641802 2566 flags.go:64] FLAG: --rotate-certificates="false" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641805 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641808 2566 flags.go:64] FLAG: --runonce="false" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641811 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641814 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641820 2566 flags.go:64] FLAG: --seccomp-default="false" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641822 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641825 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641829 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641832 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641835 2566 flags.go:64] FLAG: --storage-driver-password="root" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641838 2566 flags.go:64] FLAG: --storage-driver-secure="false" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641841 2566 flags.go:64] FLAG: --storage-driver-table="stats" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641844 2566 flags.go:64] FLAG: --storage-driver-user="root" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641847 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641850 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641855 2566 flags.go:64] FLAG: --system-cgroups="" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641871 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641877 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641880 2566 flags.go:64] FLAG: --tls-cert-file="" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641883 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641887 2566 flags.go:64] FLAG: --tls-min-version="" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641890 2566 flags.go:64] FLAG: --tls-private-key-file="" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641893 2566 flags.go:64] FLAG: --topology-manager-policy="none" Jul 13 14:20:58.650597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641896 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641899 2566 flags.go:64] FLAG: --topology-manager-scope="container" Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641902 2566 flags.go:64] FLAG: --v="2" Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641906 2566 flags.go:64] FLAG: --version="false" Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641910 2566 flags.go:64] FLAG: --vmodule="" Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641917 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.641920 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642052 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642056 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642060 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642063 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642066 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642069 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642075 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642079 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642082 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642085 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642088 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642091 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642094 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:20:58.651269 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642097 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642100 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642103 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642107 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642110 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642113 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642116 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642119 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642123 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642127 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642130 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642133 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642136 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642138 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642141 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642144 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642146 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642149 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642152 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642155 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:20:58.651776 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642157 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642160 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642163 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642166 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642169 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642173 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642176 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642178 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642181 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642184 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642187 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642189 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642192 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642195 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642198 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642202 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642205 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642207 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642211 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:20:58.652321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642213 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642216 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642218 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642221 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642224 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642227 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642229 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642232 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642237 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642240 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642244 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642247 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642250 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642253 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642256 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642259 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642261 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642264 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642268 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:20:58.652841 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642271 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642274 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642277 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642279 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642282 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642285 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642288 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642290 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642293 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642297 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642300 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642302 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642305 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642309 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642311 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642314 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642317 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642320 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642322 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642325 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:20:58.653419 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642328 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:20:58.654157 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642331 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:20:58.654157 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642334 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:20:58.654157 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642336 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:20:58.654157 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642339 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:20:58.654157 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642342 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:20:58.654157 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642345 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:20:58.654157 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.642347 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:20:58.654157 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.643358 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 14:20:58.654157 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.650535 2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 13 14:20:58.654157 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.650551 2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 13 14:20:58.654157 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.650567 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 13 14:20:58.654157 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.650574 2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 13 14:20:58.654157 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650640 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:20:58.654157 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650646 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:20:58.654157 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650650 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650654 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650657 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650660 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650663 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650665 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650668 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650671 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650674 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650677 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650688 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650691 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650693 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650696 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650699 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650702 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650704 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650707 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650710 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650713 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:20:58.654686 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650715 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650718 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650720 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650723 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650726 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650729 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650732 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650735 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650738 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650741 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650744 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650746 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650749 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650752 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650755 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650758 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650761 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650764 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650766 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650776 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:20:58.655262 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650779 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650781 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650784 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650793 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650796 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650799 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650802 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650804 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650807 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650810 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650813 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650815 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650818 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650821 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650823 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650826 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650829 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650833 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650838 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:20:58.655803 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650843 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650846 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650849 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650852 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650855 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650873 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650877 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650880 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650883 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650885 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650888 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650891 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650894 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650897 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650899 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650902 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650904 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650914 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650917 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:20:58.656324 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650921 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650924 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650928 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650931 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650934 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650937 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650940 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650943 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650946 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650949 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650951 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650954 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650957 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650960 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650962 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650965 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650968 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650970 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:20:58.656819 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.650973 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:20:58.657330 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.650978 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 14:20:58.657330 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651142 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 14:20:58.657330 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651147 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 14:20:58.657330 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651150 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 14:20:58.657330 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651153 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 14:20:58.657330 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651155 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 14:20:58.657330 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651158 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 14:20:58.657330 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651161 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 14:20:58.657330 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651163 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 14:20:58.657330 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651167 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 14:20:58.657330 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651171 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 14:20:58.657330 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651174 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 14:20:58.657330 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651183 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 14:20:58.657330 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651187 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651190 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651193 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651196 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651199 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651203 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651206 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651210 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651213 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651216 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651219 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651222 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651225 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651228 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651231 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651234 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651237 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651240 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651243 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651246 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 14:20:58.657697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651249 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651252 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651255 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651258 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651260 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651263 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651266 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651268 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651271 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651274 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651277 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651280 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651289 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651292 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651294 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651297 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651299 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651302 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651305 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651308 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 14:20:58.658310 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651310 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651313 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651316 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651318 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651321 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651324 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651327 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651329 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651332 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651335 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651338 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651340 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651343 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651362 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651366 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651370 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651372 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651375 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651379 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651382 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 14:20:58.658827 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651385 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651388 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651391 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651394 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651397 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651407 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651411 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651413 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651416 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651419 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651422 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651425 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651428 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651431 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651433 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651437 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651439 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651442 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651444 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 14:20:58.659393 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651447 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 14:20:58.659928 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651450 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 14:20:58.659928 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651452 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 14:20:58.659928 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651455 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 14:20:58.659928 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651458 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 14:20:58.659928 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651460 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 14:20:58.659928 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651463 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 14:20:58.659928 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:58.651467 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 14:20:58.659928 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.651472 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 14:20:58.659928 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.651579 2566 server.go:962] "Client rotation is on, will bootstrap in background" Jul 13 14:20:58.659928 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.655016 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 13 14:20:58.659928 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.656424 2566 server.go:1019] "Starting client certificate rotation" Jul 13 14:20:58.659928 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.656526 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 14:20:58.659928 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.656567 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 14:20:58.660337 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.660013 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 14:20:58.661205 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.661191 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 14:20:58.676574 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.676553 2566 log.go:25] "Validated CRI v1 runtime API" Jul 13 14:20:58.678667 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.678649 2566 log.go:25] "Validated CRI v1 image API" Jul 13 14:20:58.679605 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.679589 2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 13 14:20:58.684861 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.684832 2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 885dc9d4-bf3f-4e4e-90c0-0bb6f7e69d28:/dev/nvme0n1p4 f8a5db56-9e9d-493d-81a3-fcd0e979257c:/dev/nvme0n1p3] Jul 13 14:20:58.684941 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.684860 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 13 14:20:58.689491 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.689471 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 14:20:58.689780 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.689662 2566 manager.go:216] Machine: {Timestamp:2026-07-13 14:20:58.689094293 +0000 UTC m=+0.373514592 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089567 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec252ffc52e604fabf540511c1cf3fbd SystemUUID:ec252ffc-52e6-04fa-bf54-0511c1cf3fbd BootID:ff38047e-0c30-46be-ad62-21da91a2d3ff Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7a:59:1a:39:01 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7a:59:1a:39:01 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:8a:79:de:63:06 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 13 14:20:58.689780 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.689771 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 13 14:20:58.690117 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.689951 2566 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 13 14:20:58.691343 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.691317 2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 13 14:20:58.691475 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.691345 2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-188.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 13 14:20:58.692075 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.692062 2566 topology_manager.go:138] "Creating topology manager with none policy" Jul 13 14:20:58.692117 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.692078 2566 container_manager_linux.go:309] "Creating device plugin manager" Jul 13 14:20:58.692117 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.692086 2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 14:20:58.693238 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.693226 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 14:20:58.694669 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.694659 2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 13 14:20:58.698464 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.698451 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 13 14:20:58.698604 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.698593 2566 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 13 14:20:58.698649 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.698624 2566 kubelet.go:486] "Attempting to sync node with API server" Jul 13 14:20:58.698649 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.698638 2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 13 14:20:58.698729 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.698666 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 13 14:20:58.698729 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.698675 2566 kubelet.go:398] "Adding apiserver pod source" Jul 13 14:20:58.698729 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.698689 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 13 14:20:58.699182 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.699167 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 14:20:58.699256 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.699191 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 14:20:58.701469 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.701452 2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 13 14:20:58.703408 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.703391 2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 13 14:20:58.703493 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.703433 2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 13 14:20:58.703655 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.703642 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 13 14:20:58.703720 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.703662 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 13 14:20:58.703720 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.703672 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 13 14:20:58.703720 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.703680 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 13 14:20:58.703720 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.703693 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 13 14:20:58.703720 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.703702 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 13 14:20:58.704333 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.704320 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 13 14:20:58.704387 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.704335 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 13 14:20:58.704387 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.704345 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 13 14:20:58.704387 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.704354 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 13 14:20:58.704387 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.704379 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 13 14:20:58.704541 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.704432 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 13 14:20:58.705027 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.705015 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 13 14:20:58.705088 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.705030 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 13 14:20:58.709699 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.709676 2566 server.go:1268] "Started kubelet" Jul 13 14:20:58.709806 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.709737 2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 13 14:20:58.709958 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.709853 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 13 14:20:58.710012 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.709984 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 13 14:20:58.710907 ip-10-0-131-188 systemd[1]: Started Kubernetes Kubelet. Jul 13 14:20:58.711165 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.711115 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-188.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 13 14:20:58.711233 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:58.711155 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-188.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 13 14:20:58.711286 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:58.711227 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 13 14:20:58.711541 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.711496 2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 13 14:20:58.716861 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.716843 2566 server.go:310] "Adding debug handlers to kubelet server" Jul 13 14:20:58.717171 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:58.717142 2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 13 14:20:58.720838 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:58.719675 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-188.ec2.internal.18c1df4f3592700a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-188.ec2.internal,UID:ip-10-0-131-188.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-188.ec2.internal,},FirstTimestamp:2026-07-13 14:20:58.70962689 +0000 UTC m=+0.394047190,LastTimestamp:2026-07-13 14:20:58.70962689 +0000 UTC m=+0.394047190,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-188.ec2.internal,}" Jul 13 14:20:58.721960 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.721938 2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 13 14:20:58.721960 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.721951 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 13 14:20:58.722117 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:58.722087 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-188.ec2.internal\" not found" Jul 13 14:20:58.722544 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.722522 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 13 14:20:58.722666 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.722547 2566 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 13 14:20:58.722666 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.722564 2566 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 13 14:20:58.722856 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.722692 2566 reconstruct.go:97] "Volume reconstruction finished" Jul 13 14:20:58.722856 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.722701 2566 reconciler.go:29] "Reconciler: start to sync state" Jul 13 14:20:58.722856 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.722820 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j6q47" Jul 13 14:20:58.723080 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.723008 2566 factory.go:55] Registering systemd factory Jul 13 14:20:58.723080 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.723024 2566 factory.go:223] Registration of the systemd container factory successfully Jul 13 14:20:58.723256 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.723240 2566 factory.go:153] Registering CRI-O factory Jul 13 14:20:58.723256 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.723257 2566 factory.go:223] Registration of the crio container factory successfully Jul 13 14:20:58.723368 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.723295 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 13 14:20:58.723368 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.723320 2566 factory.go:103] Registering Raw factory Jul 13 14:20:58.723368 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.723333 2566 manager.go:1196] Started watching for new ooms in manager Jul 13 14:20:58.723765 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.723748 2566 manager.go:349] Starting recovery of all containers Jul 13 14:20:58.732002 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:58.731975 2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-188.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 13 14:20:58.732137 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:58.732023 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 13 14:20:58.733951 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.733934 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j6q47" Jul 13 14:20:58.734451 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.734435 2566 manager.go:354] Recovery completed Jul 13 14:20:58.739916 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.739899 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:20:58.742079 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.742058 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:20:58.742166 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.742092 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:20:58.742166 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.742103 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:20:58.742707 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.742692 2566 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 13 14:20:58.742707 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.742707 2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 13 14:20:58.742849 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.742722 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 13 14:20:58.742849 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.742765 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:20:58.743503 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.743488 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:20:58.743577 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.743519 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:20:58.743577 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.743534 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:20:58.743806 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.743791 2566 policy_none.go:49] "None policy: Start" Jul 13 14:20:58.743806 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.743817 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 13 14:20:58.743935 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.743833 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 13 14:20:58.744935 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.744923 2566 policy_none.go:47] "Start" Jul 13 14:20:58.785426 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.785409 2566 manager.go:337] "Starting Device Plugin manager" Jul 13 14:20:58.785504 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:58.785441 2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 13 14:20:58.785504 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.785466 2566 server.go:87] "Starting device plugin registration server" Jul 13 14:20:58.785734 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.785700 2566 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 13 14:20:58.785734 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.785715 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 13 14:20:58.786134 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.785872 2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 13 14:20:58.786134 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.785982 2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 13 14:20:58.786134 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.785993 2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 13 14:20:58.786419 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:58.786396 2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 13 14:20:58.786466 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:58.786448 2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-188.ec2.internal\" not found" Jul 13 14:20:58.859742 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.859708 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 13 14:20:58.860940 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.860923 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 13 14:20:58.861004 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.860973 2566 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 13 14:20:58.861042 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.861008 2566 kubelet.go:2448] "Starting kubelet main sync loop" Jul 13 14:20:58.861122 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:58.861106 2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 13 14:20:58.864300 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.864283 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:20:58.886964 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.886909 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:20:58.887810 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.887789 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:20:58.887915 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.887824 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:20:58.887915 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.887836 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:20:58.887915 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.887878 2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-188.ec2.internal" Jul 13 14:20:58.899832 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.899814 2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-188.ec2.internal" Jul 13 14:20:58.899896 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:58.899836 2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-188.ec2.internal\": node \"ip-10-0-131-188.ec2.internal\" not found" Jul 13 14:20:58.921869 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:58.921846 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-188.ec2.internal\" not found" Jul 13 14:20:58.962030 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.962004 2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-188.ec2.internal"] Jul 13 14:20:58.962127 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.962073 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:20:58.963733 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.963716 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:20:58.963804 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.963750 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:20:58.963804 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.963761 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:20:58.964911 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.964899 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:20:58.965062 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.965032 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal" Jul 13 14:20:58.965102 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.965071 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:20:58.965664 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.965648 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:20:58.965664 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.965658 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:20:58.965814 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.965677 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:20:58.965814 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.965686 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:20:58.965814 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.965690 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:20:58.965814 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.965705 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:20:58.967304 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.967287 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.ec2.internal" Jul 13 14:20:58.967389 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.967317 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 14:20:58.968083 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.968067 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasSufficientMemory" Jul 13 14:20:58.968161 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.968099 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 14:20:58.968161 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:58.968114 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeHasSufficientPID" Jul 13 14:20:58.986441 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:58.986418 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-188.ec2.internal\" not found" node="ip-10-0-131-188.ec2.internal" Jul 13 14:20:58.990539 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:58.990524 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-188.ec2.internal\" not found" node="ip-10-0-131-188.ec2.internal" Jul 13 14:20:59.022992 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:59.022968 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-188.ec2.internal\" not found" Jul 13 14:20:59.024245 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.024214 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6883fe87cd18b4634205e5c8665ef14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal\" (UID: \"c6883fe87cd18b4634205e5c8665ef14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal" Jul 13 14:20:59.024359 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.024258 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6883fe87cd18b4634205e5c8665ef14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal\" (UID: \"c6883fe87cd18b4634205e5c8665ef14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal" Jul 13 14:20:59.024359 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.024279 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/daa422171c2cfe1891a45971f28fd4ef-config\") pod \"kube-apiserver-proxy-ip-10-0-131-188.ec2.internal\" (UID: \"daa422171c2cfe1891a45971f28fd4ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.ec2.internal" Jul 13 14:20:59.123836 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:59.123808 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-188.ec2.internal\" not found" Jul 13 14:20:59.125016 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.124999 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6883fe87cd18b4634205e5c8665ef14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal\" (UID: \"c6883fe87cd18b4634205e5c8665ef14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal" Jul 13 14:20:59.125077 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.125025 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6883fe87cd18b4634205e5c8665ef14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal\" (UID: \"c6883fe87cd18b4634205e5c8665ef14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal" Jul 13 14:20:59.125077 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.125043 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/daa422171c2cfe1891a45971f28fd4ef-config\") pod \"kube-apiserver-proxy-ip-10-0-131-188.ec2.internal\" (UID: \"daa422171c2cfe1891a45971f28fd4ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.ec2.internal" Jul 13 14:20:59.125181 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.125087 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/daa422171c2cfe1891a45971f28fd4ef-config\") pod \"kube-apiserver-proxy-ip-10-0-131-188.ec2.internal\" (UID: \"daa422171c2cfe1891a45971f28fd4ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.ec2.internal" Jul 13 14:20:59.125181 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.125120 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6883fe87cd18b4634205e5c8665ef14-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal\" (UID: \"c6883fe87cd18b4634205e5c8665ef14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal" Jul 13 14:20:59.125181 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.125126 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c6883fe87cd18b4634205e5c8665ef14-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal\" (UID: \"c6883fe87cd18b4634205e5c8665ef14\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal" Jul 13 14:20:59.224218 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:59.224118 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-188.ec2.internal\" not found" Jul 13 14:20:59.292614 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.292582 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal" Jul 13 14:20:59.293760 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.293741 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.ec2.internal" Jul 13 14:20:59.324301 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:59.324263 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-188.ec2.internal\" not found" Jul 13 14:20:59.429431 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:59.429377 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-188.ec2.internal\" not found" Jul 13 14:20:59.529498 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:59.529436 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-188.ec2.internal\" not found" Jul 13 14:20:59.629575 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:20:59.629524 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-188.ec2.internal\" not found" Jul 13 14:20:59.656730 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.656705 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 13 14:20:59.656885 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.656865 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 14:20:59.693768 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.693741 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:20:59.724072 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.724051 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:20:59.739855 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.739819 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 14:15:59 +0000 UTC" deadline="2028-04-24 23:41:33.688758162 +0000 UTC" Jul 13 14:20:59.739855 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.739848 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15633h20m33.948914432s" Jul 13 14:20:59.792280 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.792221 2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:20:59.823624 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.823596 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal" Jul 13 14:20:59.838118 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.838093 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 14:20:59.839403 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.839388 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.ec2.internal" Jul 13 14:20:59.848257 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.848094 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 14:20:59.987321 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:20:59.987280 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa422171c2cfe1891a45971f28fd4ef.slice/crio-ab99097cf42ae812a4b291c26251eb6d6f41bc492350fe91eb9be460e06bd7dd WatchSource:0}: Error finding container ab99097cf42ae812a4b291c26251eb6d6f41bc492350fe91eb9be460e06bd7dd: Status 404 returned error can't find the container with id ab99097cf42ae812a4b291c26251eb6d6f41bc492350fe91eb9be460e06bd7dd Jul 13 14:20:59.991290 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:20:59.991276 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 14:21:00.700334 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.700300 2566 apiserver.go:52] "Watching apiserver" Jul 13 14:21:00.708965 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.708942 2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 13 14:21:00.709983 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.709952 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-bmrdc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84","openshift-dns/node-resolver-xcv69","openshift-image-registry/node-ca-76bmg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal","openshift-multus/multus-hfmt7","openshift-network-diagnostics/network-check-target-sl54c","kube-system/kube-apiserver-proxy-ip-10-0-131-188.ec2.internal","openshift-cluster-node-tuning-operator/tuned-r9qr6","openshift-multus/multus-additional-cni-plugins-qbm2b","openshift-multus/network-metrics-daemon-cj24h","openshift-network-operator/iptables-alerter-lvs6p","openshift-ovn-kubernetes/ovnkube-node-pr6tf","kube-system/global-pull-secret-syncer-f9wtx"] Jul 13 14:21:00.711716 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.711689 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:00.711840 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:00.711776 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:00.713028 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.713005 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:00.713128 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:00.713082 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:00.714416 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.714395 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xcv69" Jul 13 14:21:00.715908 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.715825 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-76bmg" Jul 13 14:21:00.717343 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.717323 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 13 14:21:00.717441 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.717365 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p9zfj\"" Jul 13 14:21:00.717441 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.717380 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:00.717559 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:00.717445 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:00.717559 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.717547 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 13 14:21:00.718864 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.718838 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.718958 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.718936 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 13 14:21:00.719326 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.719302 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 13 14:21:00.719450 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.719429 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 13 14:21:00.719545 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.719527 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pbz8z\"" Jul 13 14:21:00.720334 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.720314 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.722610 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.722589 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.723551 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.723099 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 13 14:21:00.725866 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.725842 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 13 14:21:00.726212 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.726193 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 13 14:21:00.726391 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.726368 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 13 14:21:00.726494 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.726476 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 13 14:21:00.726755 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.726736 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 13 14:21:00.726930 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.726909 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 13 14:21:00.727025 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.727008 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 13 14:21:00.727406 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.727385 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.727514 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.727496 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dnxgn\"" Jul 13 14:21:00.727695 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.727671 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-d47gv\"" Jul 13 14:21:00.727915 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.727388 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 13 14:21:00.728340 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.728318 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 13 14:21:00.728562 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.728542 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4nsxt\"" Jul 13 14:21:00.728646 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.728628 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 13 14:21:00.728880 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.728805 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 13 14:21:00.728961 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.728910 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 13 14:21:00.729400 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.729375 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.730947 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.730927 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bmrdc" Jul 13 14:21:00.732509 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.732486 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2l85v\"" Jul 13 14:21:00.732601 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.732492 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lvs6p" Jul 13 14:21:00.735586 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.735559 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 13 14:21:00.735683 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.735593 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 13 14:21:00.735683 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.735625 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 13 14:21:00.735945 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.735923 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 13 14:21:00.736023 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.735957 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 13 14:21:00.736762 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.736741 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h2vrm\"" Jul 13 14:21:00.736875 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.736860 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-var-lib-cni-multus\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.736935 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.736893 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-var-lib-kubelet\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.736935 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.736896 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jt5d6\"" Jul 13 14:21:00.736935 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.736923 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.737089 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.736898 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 13 14:21:00.737089 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.736986 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-etc-selinux\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.737089 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737016 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p2lm8\"" Jul 13 14:21:00.737089 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737039 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-sys-fs\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.737089 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737077 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-var-lib-cni-bin\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.737315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737115 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-run-openvswitch\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.737315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737133 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 13 14:21:00.737315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737144 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-run-multus-certs\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.737315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737171 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pblgn\" (UniqueName: \"kubernetes.io/projected/cde58de7-b1b7-41c5-83c1-eaed967537b2-kube-api-access-pblgn\") pod \"network-metrics-daemon-cj24h\" (UID: \"cde58de7-b1b7-41c5-83c1-eaed967537b2\") " pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:00.737315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737196 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-systemd-units\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.737315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737219 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-slash\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.737315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737242 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-run-netns\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.737315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737264 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-run-ovn\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.737315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737287 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-log-socket\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.737315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737312 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-cni-netd\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.737788 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737364 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7rj9\" (UniqueName: \"kubernetes.io/projected/5962361d-1783-4050-8470-a1b9518399bb-kube-api-access-d7rj9\") pod \"node-ca-76bmg\" (UID: \"5962361d-1783-4050-8470-a1b9518399bb\") " pod="openshift-image-registry/node-ca-76bmg" Jul 13 14:21:00.737788 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737393 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-device-dir\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.737788 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737434 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-kubelet-config\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:00.737788 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737491 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-dbus\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:00.737788 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737513 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11a8e255-d719-4783-be00-d89aca5244c5-cni-binary-copy\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.737788 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737535 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-multus-conf-dir\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.737788 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737559 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fcd601a1-5d41-4039-bdb4-219a3e4aef87-ovnkube-config\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.737788 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737594 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-multus-socket-dir-parent\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.737788 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737626 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-host\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.737788 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737651 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-node-log\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.737788 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737675 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-registration-dir\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.737788 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737698 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-multus-cni-dir\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.737788 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737737 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-var-lib-kubelet\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.737788 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737760 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-modprobe-d\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737834 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-sys\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737868 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b676g\" (UniqueName: \"kubernetes.io/projected/663d50ff-7b28-4a5e-a211-6761789a969c-kube-api-access-b676g\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737911 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgkkb\" (UniqueName: \"kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb\") pod \"network-check-target-sl54c\" (UID: \"4629c8d6-afb7-43da-b676-9d6418414491\") " pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737936 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-run-systemd\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737958 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-run-ovn-kubernetes\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.737997 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-system-cni-dir\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738030 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-run-k8s-cni-cncf-io\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738060 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-hostroot\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738096 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-sysctl-conf\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738120 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fcd601a1-5d41-4039-bdb4-219a3e4aef87-ovnkube-script-lib\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738144 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-cni-bin\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738167 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-socket-dir\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738192 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blc8z\" (UniqueName: \"kubernetes.io/projected/d5ab8d67-8d30-4d28-be21-8210bedfea5b-kube-api-access-blc8z\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738195 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738246 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11a8e255-d719-4783-be00-d89aca5244c5-multus-daemon-config\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738281 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-etc-kubernetes\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.738529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738308 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-run\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738332 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91eed05b-8ccb-4806-ac18-2ced768c9780-tmp-dir\") pod \"node-resolver-xcv69\" (UID: \"91eed05b-8ccb-4806-ac18-2ced768c9780\") " pod="openshift-dns/node-resolver-xcv69" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738358 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfsvv\" (UniqueName: \"kubernetes.io/projected/91eed05b-8ccb-4806-ac18-2ced768c9780-kube-api-access-sfsvv\") pod \"node-resolver-xcv69\" (UID: \"91eed05b-8ccb-4806-ac18-2ced768c9780\") " pod="openshift-dns/node-resolver-xcv69" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738381 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5962361d-1783-4050-8470-a1b9518399bb-host\") pod \"node-ca-76bmg\" (UID: \"5962361d-1783-4050-8470-a1b9518399bb\") " pod="openshift-image-registry/node-ca-76bmg" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738405 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738453 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-sysctl-d\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738498 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-lib-modules\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738523 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/663d50ff-7b28-4a5e-a211-6761789a969c-tmp\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738551 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-var-lib-openvswitch\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738578 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-etc-openvswitch\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738611 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdftt\" (UniqueName: \"kubernetes.io/projected/fcd601a1-5d41-4039-bdb4-219a3e4aef87-kube-api-access-tdftt\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738638 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91eed05b-8ccb-4806-ac18-2ced768c9780-hosts-file\") pod \"node-resolver-xcv69\" (UID: \"91eed05b-8ccb-4806-ac18-2ced768c9780\") " pod="openshift-dns/node-resolver-xcv69" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738661 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5962361d-1783-4050-8470-a1b9518399bb-serviceca\") pod \"node-ca-76bmg\" (UID: \"5962361d-1783-4050-8470-a1b9518399bb\") " pod="openshift-image-registry/node-ca-76bmg" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738682 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-kubernetes\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738708 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-os-release\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738733 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xp57\" (UniqueName: \"kubernetes.io/projected/11a8e255-d719-4783-be00-d89aca5244c5-kube-api-access-6xp57\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738758 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.739367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738782 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fcd601a1-5d41-4039-bdb4-219a3e4aef87-env-overrides\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.740209 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738789 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 13 14:21:00.740209 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738837 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/663d50ff-7b28-4a5e-a211-6761789a969c-etc-tuned\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.740209 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738862 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-kubelet\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.740209 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738886 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-cnibin\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.740209 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738909 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-run-netns\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.740209 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738935 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-sysconfig\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.740209 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738959 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-systemd\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.740209 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.738998 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fcd601a1-5d41-4039-bdb4-219a3e4aef87-ovn-node-metrics-cert\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.740209 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.739040 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs\") pod \"network-metrics-daemon-cj24h\" (UID: \"cde58de7-b1b7-41c5-83c1-eaed967537b2\") " pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:00.740209 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.739518 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 13 14:21:00.744719 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.744700 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 14:21:00.767531 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.767508 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wwmmm" Jul 13 14:21:00.777425 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.777404 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wwmmm" Jul 13 14:21:00.823704 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.823680 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 13 14:21:00.839250 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839228 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-var-lib-kubelet\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.839312 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839254 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-modprobe-d\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.839312 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839270 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-sys\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.839312 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839285 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b676g\" (UniqueName: \"kubernetes.io/projected/663d50ff-7b28-4a5e-a211-6761789a969c-kube-api-access-b676g\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.839427 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839316 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkkb\" (UniqueName: \"kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb\") pod \"network-check-target-sl54c\" (UID: \"4629c8d6-afb7-43da-b676-9d6418414491\") " pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:00.839427 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839337 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-run-systemd\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.839427 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839355 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-var-lib-kubelet\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.839427 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839390 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-sys\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.839427 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839409 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-modprobe-d\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.839427 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839395 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-run-ovn-kubernetes\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.839634 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839428 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-run-ovn-kubernetes\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.839634 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839470 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-run-systemd\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.839634 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839589 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-system-cni-dir\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.839634 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839620 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-run-k8s-cni-cncf-io\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.839770 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839638 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-hostroot\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.839770 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839654 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-sysctl-conf\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.839770 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839684 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-run-k8s-cni-cncf-io\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.839770 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839712 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fcd601a1-5d41-4039-bdb4-219a3e4aef87-ovnkube-script-lib\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.839770 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839713 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-hostroot\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.839770 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839719 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-system-cni-dir\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.839770 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839731 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-cni-bin\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.839770 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839753 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-cni-bin\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.839770 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839764 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-socket-dir\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839768 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-sysctl-conf\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839793 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blc8z\" (UniqueName: \"kubernetes.io/projected/d5ab8d67-8d30-4d28-be21-8210bedfea5b-kube-api-access-blc8z\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839852 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11a8e255-d719-4783-be00-d89aca5244c5-multus-daemon-config\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839876 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-etc-kubernetes\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839896 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-run\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839930 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91eed05b-8ccb-4806-ac18-2ced768c9780-tmp-dir\") pod \"node-resolver-xcv69\" (UID: \"91eed05b-8ccb-4806-ac18-2ced768c9780\") " pod="openshift-dns/node-resolver-xcv69" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839954 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfsvv\" (UniqueName: \"kubernetes.io/projected/91eed05b-8ccb-4806-ac18-2ced768c9780-kube-api-access-sfsvv\") pod \"node-resolver-xcv69\" (UID: \"91eed05b-8ccb-4806-ac18-2ced768c9780\") " pod="openshift-dns/node-resolver-xcv69" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839969 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5962361d-1783-4050-8470-a1b9518399bb-host\") pod \"node-ca-76bmg\" (UID: \"5962361d-1783-4050-8470-a1b9518399bb\") " pod="openshift-image-registry/node-ca-76bmg" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.839993 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840030 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-etc-kubernetes\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840039 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5962361d-1783-4050-8470-a1b9518399bb-host\") pod \"node-ca-76bmg\" (UID: \"5962361d-1783-4050-8470-a1b9518399bb\") " pod="openshift-image-registry/node-ca-76bmg" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840067 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-sysctl-d\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840087 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-run\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840092 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-lib-modules\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840123 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/663d50ff-7b28-4a5e-a211-6761789a969c-tmp\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.840140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840149 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-var-lib-openvswitch\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840176 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-etc-openvswitch\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840187 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-lib-modules\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840203 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdftt\" (UniqueName: \"kubernetes.io/projected/fcd601a1-5d41-4039-bdb4-219a3e4aef87-kube-api-access-tdftt\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840227 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91eed05b-8ccb-4806-ac18-2ced768c9780-hosts-file\") pod \"node-resolver-xcv69\" (UID: \"91eed05b-8ccb-4806-ac18-2ced768c9780\") " pod="openshift-dns/node-resolver-xcv69" Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840265 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-var-lib-openvswitch\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840066 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-socket-dir\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:00.840229 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840316 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-sysctl-d\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840347 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fcd601a1-5d41-4039-bdb4-219a3e4aef87-ovnkube-script-lib\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840367 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-etc-openvswitch\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840287 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5962361d-1783-4050-8470-a1b9518399bb-serviceca\") pod \"node-ca-76bmg\" (UID: \"5962361d-1783-4050-8470-a1b9518399bb\") " pod="openshift-image-registry/node-ca-76bmg" Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840267 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91eed05b-8ccb-4806-ac18-2ced768c9780-tmp-dir\") pod \"node-resolver-xcv69\" (UID: \"91eed05b-8ccb-4806-ac18-2ced768c9780\") " pod="openshift-dns/node-resolver-xcv69" Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:00.840410 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret podName:1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b nodeName:}" failed. No retries permitted until 2026-07-13 14:21:01.340367269 +0000 UTC m=+3.024787572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret") pod "global-pull-secret-syncer-f9wtx" (UID: "1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840458 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-kubernetes\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840475 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11a8e255-d719-4783-be00-d89aca5244c5-multus-daemon-config\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840490 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjrjf\" (UniqueName: \"kubernetes.io/projected/a0d488fc-58f7-409e-ba15-7f7980cbd394-kube-api-access-vjrjf\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.840956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840539 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-kubernetes\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840539 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91eed05b-8ccb-4806-ac18-2ced768c9780-hosts-file\") pod \"node-resolver-xcv69\" (UID: \"91eed05b-8ccb-4806-ac18-2ced768c9780\") " pod="openshift-dns/node-resolver-xcv69" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840565 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/58d081b3-2720-409f-8b1c-a82bc7238691-iptables-alerter-script\") pod \"iptables-alerter-lvs6p\" (UID: \"58d081b3-2720-409f-8b1c-a82bc7238691\") " pod="openshift-network-operator/iptables-alerter-lvs6p" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840594 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58d081b3-2720-409f-8b1c-a82bc7238691-host-slash\") pod \"iptables-alerter-lvs6p\" (UID: \"58d081b3-2720-409f-8b1c-a82bc7238691\") " pod="openshift-network-operator/iptables-alerter-lvs6p" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840619 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-os-release\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840594 2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840642 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xp57\" (UniqueName: \"kubernetes.io/projected/11a8e255-d719-4783-be00-d89aca5244c5-kube-api-access-6xp57\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840665 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840674 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-os-release\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840682 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0d488fc-58f7-409e-ba15-7f7980cbd394-cnibin\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840748 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840800 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fcd601a1-5d41-4039-bdb4-219a3e4aef87-env-overrides\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840806 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5962361d-1783-4050-8470-a1b9518399bb-serviceca\") pod \"node-ca-76bmg\" (UID: \"5962361d-1783-4050-8470-a1b9518399bb\") " pod="openshift-image-registry/node-ca-76bmg" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840825 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/abb92db6-649f-49f1-8ec1-5f41c88f9cfb-konnectivity-ca\") pod \"konnectivity-agent-bmrdc\" (UID: \"abb92db6-649f-49f1-8ec1-5f41c88f9cfb\") " pod="kube-system/konnectivity-agent-bmrdc" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840850 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ftrz\" (UniqueName: \"kubernetes.io/projected/58d081b3-2720-409f-8b1c-a82bc7238691-kube-api-access-8ftrz\") pod \"iptables-alerter-lvs6p\" (UID: \"58d081b3-2720-409f-8b1c-a82bc7238691\") " pod="openshift-network-operator/iptables-alerter-lvs6p" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840877 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/663d50ff-7b28-4a5e-a211-6761789a969c-etc-tuned\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840900 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-kubelet\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840923 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/abb92db6-649f-49f1-8ec1-5f41c88f9cfb-agent-certs\") pod \"konnectivity-agent-bmrdc\" (UID: \"abb92db6-649f-49f1-8ec1-5f41c88f9cfb\") " pod="kube-system/konnectivity-agent-bmrdc" Jul 13 14:21:00.841659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840949 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-cnibin\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.840957 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-kubelet\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841434 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-run-netns\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841467 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-sysconfig\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841498 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-systemd\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841530 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fcd601a1-5d41-4039-bdb4-219a3e4aef87-ovn-node-metrics-cert\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841574 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs\") pod \"network-metrics-daemon-cj24h\" (UID: \"cde58de7-b1b7-41c5-83c1-eaed967537b2\") " pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841612 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a0d488fc-58f7-409e-ba15-7f7980cbd394-os-release\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841615 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fcd601a1-5d41-4039-bdb4-219a3e4aef87-env-overrides\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841656 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a0d488fc-58f7-409e-ba15-7f7980cbd394-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841729 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-cnibin\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841730 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-run-netns\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841770 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-var-lib-cni-multus\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841789 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-systemd\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841847 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-etc-sysconfig\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841913 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-var-lib-kubelet\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841940 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-var-lib-kubelet\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.842507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.841954 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842001 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842016 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-etc-selinux\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842018 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-var-lib-cni-multus\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:00.842096 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842117 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-etc-selinux\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842191 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-sys-fs\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842242 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-var-lib-cni-bin\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842280 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-run-openvswitch\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:00.842324 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs podName:cde58de7-b1b7-41c5-83c1-eaed967537b2 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:01.342297333 +0000 UTC m=+3.026717634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs") pod "network-metrics-daemon-cj24h" (UID: "cde58de7-b1b7-41c5-83c1-eaed967537b2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842341 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-run-openvswitch\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842355 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-sys-fs\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842414 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-var-lib-cni-bin\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842455 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-run-multus-certs\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842497 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pblgn\" (UniqueName: \"kubernetes.io/projected/cde58de7-b1b7-41c5-83c1-eaed967537b2-kube-api-access-pblgn\") pod \"network-metrics-daemon-cj24h\" (UID: \"cde58de7-b1b7-41c5-83c1-eaed967537b2\") " pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842525 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-systemd-units\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842557 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-slash\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.843396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842568 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-host-run-multus-certs\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842586 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-run-netns\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842635 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-run-netns\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842675 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-run-ovn\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842725 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-run-ovn\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842757 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-log-socket\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842814 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-log-socket\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842850 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-cni-netd\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842899 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7rj9\" (UniqueName: \"kubernetes.io/projected/5962361d-1783-4050-8470-a1b9518399bb-kube-api-access-d7rj9\") pod \"node-ca-76bmg\" (UID: \"5962361d-1783-4050-8470-a1b9518399bb\") " pod="openshift-image-registry/node-ca-76bmg" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842912 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-slash\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842939 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-device-dir\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842973 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-kubelet-config\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.842998 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-host-cni-netd\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.843004 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-dbus\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.843060 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11a8e255-d719-4783-be00-d89aca5244c5-cni-binary-copy\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.843066 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-dbus\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.843159 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-systemd-units\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.843225 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-device-dir\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.844263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.843285 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-kubelet-config\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.843406 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-multus-conf-dir\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.843455 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fcd601a1-5d41-4039-bdb4-219a3e4aef87-ovnkube-config\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.843498 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0d488fc-58f7-409e-ba15-7f7980cbd394-cni-binary-copy\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.843858 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-multus-conf-dir\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.843914 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-multus-socket-dir-parent\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.843946 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-host\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.843983 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-node-log\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.844017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-registration-dir\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.844056 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0d488fc-58f7-409e-ba15-7f7980cbd394-system-cni-dir\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.844093 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0d488fc-58f7-409e-ba15-7f7980cbd394-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.844121 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a0d488fc-58f7-409e-ba15-7f7980cbd394-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.844160 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-multus-cni-dir\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.844290 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-multus-cni-dir\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.844362 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fcd601a1-5d41-4039-bdb4-219a3e4aef87-node-log\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.844434 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5ab8d67-8d30-4d28-be21-8210bedfea5b-registration-dir\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.844520 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/663d50ff-7b28-4a5e-a211-6761789a969c-etc-tuned\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.845099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.844686 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/663d50ff-7b28-4a5e-a211-6761789a969c-tmp\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.845901 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.844762 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11a8e255-d719-4783-be00-d89aca5244c5-multus-socket-dir-parent\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.845901 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.844959 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fcd601a1-5d41-4039-bdb4-219a3e4aef87-ovnkube-config\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.845901 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.845149 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fcd601a1-5d41-4039-bdb4-219a3e4aef87-ovn-node-metrics-cert\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.845901 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.845239 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/663d50ff-7b28-4a5e-a211-6761789a969c-host\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.845901 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.845412 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11a8e255-d719-4783-be00-d89aca5244c5-cni-binary-copy\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.852690 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:00.852670 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:21:00.852790 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:00.852694 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:21:00.852790 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:00.852707 2566 projected.go:196] Error preparing data for projected volume kube-api-access-jgkkb for pod openshift-network-diagnostics/network-check-target-sl54c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:21:00.852790 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:00.852778 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb podName:4629c8d6-afb7-43da-b676-9d6418414491 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:01.352759174 +0000 UTC m=+3.037179474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jgkkb" (UniqueName: "kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb") pod "network-check-target-sl54c" (UID: "4629c8d6-afb7-43da-b676-9d6418414491") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:21:00.853759 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.853733 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7rj9\" (UniqueName: \"kubernetes.io/projected/5962361d-1783-4050-8470-a1b9518399bb-kube-api-access-d7rj9\") pod \"node-ca-76bmg\" (UID: \"5962361d-1783-4050-8470-a1b9518399bb\") " pod="openshift-image-registry/node-ca-76bmg" Jul 13 14:21:00.854340 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.854316 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xp57\" (UniqueName: \"kubernetes.io/projected/11a8e255-d719-4783-be00-d89aca5244c5-kube-api-access-6xp57\") pod \"multus-hfmt7\" (UID: \"11a8e255-d719-4783-be00-d89aca5244c5\") " pod="openshift-multus/multus-hfmt7" Jul 13 14:21:00.855212 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.855175 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b676g\" (UniqueName: \"kubernetes.io/projected/663d50ff-7b28-4a5e-a211-6761789a969c-kube-api-access-b676g\") pod \"tuned-r9qr6\" (UID: \"663d50ff-7b28-4a5e-a211-6761789a969c\") " pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:00.855335 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.855324 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdftt\" (UniqueName: \"kubernetes.io/projected/fcd601a1-5d41-4039-bdb4-219a3e4aef87-kube-api-access-tdftt\") pod \"ovnkube-node-pr6tf\" (UID: \"fcd601a1-5d41-4039-bdb4-219a3e4aef87\") " pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:00.856747 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.856719 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blc8z\" (UniqueName: \"kubernetes.io/projected/d5ab8d67-8d30-4d28-be21-8210bedfea5b-kube-api-access-blc8z\") pod \"aws-ebs-csi-driver-node-f6b84\" (UID: \"d5ab8d67-8d30-4d28-be21-8210bedfea5b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:00.856822 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.856749 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pblgn\" (UniqueName: \"kubernetes.io/projected/cde58de7-b1b7-41c5-83c1-eaed967537b2-kube-api-access-pblgn\") pod \"network-metrics-daemon-cj24h\" (UID: \"cde58de7-b1b7-41c5-83c1-eaed967537b2\") " pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:00.856822 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.856791 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfsvv\" (UniqueName: \"kubernetes.io/projected/91eed05b-8ccb-4806-ac18-2ced768c9780-kube-api-access-sfsvv\") pod \"node-resolver-xcv69\" (UID: \"91eed05b-8ccb-4806-ac18-2ced768c9780\") " pod="openshift-dns/node-resolver-xcv69" Jul 13 14:21:00.865942 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.865876 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.ec2.internal" event={"ID":"daa422171c2cfe1891a45971f28fd4ef","Type":"ContainerStarted","Data":"ab99097cf42ae812a4b291c26251eb6d6f41bc492350fe91eb9be460e06bd7dd"} Jul 13 14:21:00.931516 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.931489 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 14:21:00.945251 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945222 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0d488fc-58f7-409e-ba15-7f7980cbd394-cni-binary-copy\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.945390 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945369 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0d488fc-58f7-409e-ba15-7f7980cbd394-system-cni-dir\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.945460 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945404 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0d488fc-58f7-409e-ba15-7f7980cbd394-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.945460 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945430 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a0d488fc-58f7-409e-ba15-7f7980cbd394-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.945606 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945541 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjrjf\" (UniqueName: \"kubernetes.io/projected/a0d488fc-58f7-409e-ba15-7f7980cbd394-kube-api-access-vjrjf\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.945717 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945597 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0d488fc-58f7-409e-ba15-7f7980cbd394-system-cni-dir\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.945717 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945603 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0d488fc-58f7-409e-ba15-7f7980cbd394-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.945717 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945625 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/58d081b3-2720-409f-8b1c-a82bc7238691-iptables-alerter-script\") pod \"iptables-alerter-lvs6p\" (UID: \"58d081b3-2720-409f-8b1c-a82bc7238691\") " pod="openshift-network-operator/iptables-alerter-lvs6p" Jul 13 14:21:00.945717 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945653 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58d081b3-2720-409f-8b1c-a82bc7238691-host-slash\") pod \"iptables-alerter-lvs6p\" (UID: \"58d081b3-2720-409f-8b1c-a82bc7238691\") " pod="openshift-network-operator/iptables-alerter-lvs6p" Jul 13 14:21:00.945943 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945809 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0d488fc-58f7-409e-ba15-7f7980cbd394-cnibin\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.945943 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945840 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/abb92db6-649f-49f1-8ec1-5f41c88f9cfb-konnectivity-ca\") pod \"konnectivity-agent-bmrdc\" (UID: \"abb92db6-649f-49f1-8ec1-5f41c88f9cfb\") " pod="kube-system/konnectivity-agent-bmrdc" Jul 13 14:21:00.945943 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945876 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ftrz\" (UniqueName: \"kubernetes.io/projected/58d081b3-2720-409f-8b1c-a82bc7238691-kube-api-access-8ftrz\") pod \"iptables-alerter-lvs6p\" (UID: \"58d081b3-2720-409f-8b1c-a82bc7238691\") " pod="openshift-network-operator/iptables-alerter-lvs6p" Jul 13 14:21:00.945943 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945904 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0d488fc-58f7-409e-ba15-7f7980cbd394-cnibin\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.945943 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945914 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0d488fc-58f7-409e-ba15-7f7980cbd394-cni-binary-copy\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.945943 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945941 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/58d081b3-2720-409f-8b1c-a82bc7238691-host-slash\") pod \"iptables-alerter-lvs6p\" (UID: \"58d081b3-2720-409f-8b1c-a82bc7238691\") " pod="openshift-network-operator/iptables-alerter-lvs6p" Jul 13 14:21:00.946255 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945905 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/abb92db6-649f-49f1-8ec1-5f41c88f9cfb-agent-certs\") pod \"konnectivity-agent-bmrdc\" (UID: \"abb92db6-649f-49f1-8ec1-5f41c88f9cfb\") " pod="kube-system/konnectivity-agent-bmrdc" Jul 13 14:21:00.946255 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.945992 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a0d488fc-58f7-409e-ba15-7f7980cbd394-os-release\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.946255 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.946012 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a0d488fc-58f7-409e-ba15-7f7980cbd394-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.946255 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.946188 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a0d488fc-58f7-409e-ba15-7f7980cbd394-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.946255 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.946218 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/58d081b3-2720-409f-8b1c-a82bc7238691-iptables-alerter-script\") pod \"iptables-alerter-lvs6p\" (UID: \"58d081b3-2720-409f-8b1c-a82bc7238691\") " pod="openshift-network-operator/iptables-alerter-lvs6p" Jul 13 14:21:00.946457 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.946301 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a0d488fc-58f7-409e-ba15-7f7980cbd394-os-release\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.946512 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.946457 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/abb92db6-649f-49f1-8ec1-5f41c88f9cfb-konnectivity-ca\") pod \"konnectivity-agent-bmrdc\" (UID: \"abb92db6-649f-49f1-8ec1-5f41c88f9cfb\") " pod="kube-system/konnectivity-agent-bmrdc" Jul 13 14:21:00.946567 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.946512 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a0d488fc-58f7-409e-ba15-7f7980cbd394-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:00.948542 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.948520 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/abb92db6-649f-49f1-8ec1-5f41c88f9cfb-agent-certs\") pod \"konnectivity-agent-bmrdc\" (UID: \"abb92db6-649f-49f1-8ec1-5f41c88f9cfb\") " pod="kube-system/konnectivity-agent-bmrdc" Jul 13 14:21:00.959434 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.959367 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ftrz\" (UniqueName: \"kubernetes.io/projected/58d081b3-2720-409f-8b1c-a82bc7238691-kube-api-access-8ftrz\") pod \"iptables-alerter-lvs6p\" (UID: \"58d081b3-2720-409f-8b1c-a82bc7238691\") " pod="openshift-network-operator/iptables-alerter-lvs6p" Jul 13 14:21:00.959571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:00.959546 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjrjf\" (UniqueName: \"kubernetes.io/projected/a0d488fc-58f7-409e-ba15-7f7980cbd394-kube-api-access-vjrjf\") pod \"multus-additional-cni-plugins-qbm2b\" (UID: \"a0d488fc-58f7-409e-ba15-7f7980cbd394\") " pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:01.028458 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.028424 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xcv69" Jul 13 14:21:01.035765 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.035741 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-76bmg" Jul 13 14:21:01.039651 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:01.039620 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91eed05b_8ccb_4806_ac18_2ced768c9780.slice/crio-020b4532afd7de4d66f4a3fc5fc00efc1b7055fb836a5801652a8c8886df7d86 WatchSource:0}: Error finding container 020b4532afd7de4d66f4a3fc5fc00efc1b7055fb836a5801652a8c8886df7d86: Status 404 returned error can't find the container with id 020b4532afd7de4d66f4a3fc5fc00efc1b7055fb836a5801652a8c8886df7d86 Jul 13 14:21:01.044805 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:01.044777 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5962361d_1783_4050_8470_a1b9518399bb.slice/crio-61a960c1654abb526b3f4d55800ac63092d577e0c7bab21339b5a50c53f303d2 WatchSource:0}: Error finding container 61a960c1654abb526b3f4d55800ac63092d577e0c7bab21339b5a50c53f303d2: Status 404 returned error can't find the container with id 61a960c1654abb526b3f4d55800ac63092d577e0c7bab21339b5a50c53f303d2 Jul 13 14:21:01.045948 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.045929 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hfmt7" Jul 13 14:21:01.052808 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.052779 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" Jul 13 14:21:01.056089 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:01.056061 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a8e255_d719_4783_be00_d89aca5244c5.slice/crio-852de9a8acec450a733f5ae000ea57599d54bcd19fc66a6ef8ba8ba988ba5774 WatchSource:0}: Error finding container 852de9a8acec450a733f5ae000ea57599d54bcd19fc66a6ef8ba8ba988ba5774: Status 404 returned error can't find the container with id 852de9a8acec450a733f5ae000ea57599d54bcd19fc66a6ef8ba8ba988ba5774 Jul 13 14:21:01.061855 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.061796 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:01.063471 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:01.063452 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663d50ff_7b28_4a5e_a211_6761789a969c.slice/crio-e33954852a439e0d43aa9a3f27c56472d59cf980218b48ea2192a1435e5be402 WatchSource:0}: Error finding container e33954852a439e0d43aa9a3f27c56472d59cf980218b48ea2192a1435e5be402: Status 404 returned error can't find the container with id e33954852a439e0d43aa9a3f27c56472d59cf980218b48ea2192a1435e5be402 Jul 13 14:21:01.068475 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.068414 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" Jul 13 14:21:01.071513 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:01.071490 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd601a1_5d41_4039_bdb4_219a3e4aef87.slice/crio-82c2ff098a8101df05062888dfb69cf0c27c1cb8abbd80c21a04944778e33e83 WatchSource:0}: Error finding container 82c2ff098a8101df05062888dfb69cf0c27c1cb8abbd80c21a04944778e33e83: Status 404 returned error can't find the container with id 82c2ff098a8101df05062888dfb69cf0c27c1cb8abbd80c21a04944778e33e83 Jul 13 14:21:01.076437 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.076399 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qbm2b" Jul 13 14:21:01.083100 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.083073 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bmrdc" Jul 13 14:21:01.086929 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:01.086892 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d488fc_58f7_409e_ba15_7f7980cbd394.slice/crio-fb4c801a6f5a22dfd19ed0ac76827c62f0572538e5838af7fee29ee56fd298f1 WatchSource:0}: Error finding container fb4c801a6f5a22dfd19ed0ac76827c62f0572538e5838af7fee29ee56fd298f1: Status 404 returned error can't find the container with id fb4c801a6f5a22dfd19ed0ac76827c62f0572538e5838af7fee29ee56fd298f1 Jul 13 14:21:01.091655 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.091635 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lvs6p" Jul 13 14:21:01.092140 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:01.092114 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb92db6_649f_49f1_8ec1_5f41c88f9cfb.slice/crio-fb26de6bed6c6931f15fc09b582826bf510dcabc53cbae402b5a25f32124f667 WatchSource:0}: Error finding container fb26de6bed6c6931f15fc09b582826bf510dcabc53cbae402b5a25f32124f667: Status 404 returned error can't find the container with id fb26de6bed6c6931f15fc09b582826bf510dcabc53cbae402b5a25f32124f667 Jul 13 14:21:01.101517 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:01.101493 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d081b3_2720_409f_8b1c_a82bc7238691.slice/crio-bb123097521692de8c9804bff31477ddd524162d91e49ce23386a39f639fab02 WatchSource:0}: Error finding container bb123097521692de8c9804bff31477ddd524162d91e49ce23386a39f639fab02: Status 404 returned error can't find the container with id bb123097521692de8c9804bff31477ddd524162d91e49ce23386a39f639fab02 Jul 13 14:21:01.348611 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.348512 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:01.348611 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.348575 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs\") pod \"network-metrics-daemon-cj24h\" (UID: \"cde58de7-b1b7-41c5-83c1-eaed967537b2\") " pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:01.348854 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:01.348693 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:21:01.348854 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:01.348698 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:21:01.348854 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:01.348761 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs podName:cde58de7-b1b7-41c5-83c1-eaed967537b2 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:02.348743945 +0000 UTC m=+4.033164245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs") pod "network-metrics-daemon-cj24h" (UID: "cde58de7-b1b7-41c5-83c1-eaed967537b2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:21:01.348854 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:01.348780 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret podName:1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b nodeName:}" failed. No retries permitted until 2026-07-13 14:21:02.348770507 +0000 UTC m=+4.033190796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret") pod "global-pull-secret-syncer-f9wtx" (UID: "1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:21:01.449360 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.449318 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkkb\" (UniqueName: \"kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb\") pod \"network-check-target-sl54c\" (UID: \"4629c8d6-afb7-43da-b676-9d6418414491\") " pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:01.449569 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:01.449544 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:21:01.449635 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:01.449579 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:21:01.449635 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:01.449594 2566 projected.go:196] Error preparing data for projected volume kube-api-access-jgkkb for pod openshift-network-diagnostics/network-check-target-sl54c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:21:01.449744 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:01.449665 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb podName:4629c8d6-afb7-43da-b676-9d6418414491 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:02.449644306 +0000 UTC m=+4.134064597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgkkb" (UniqueName: "kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb") pod "network-check-target-sl54c" (UID: "4629c8d6-afb7-43da-b676-9d6418414491") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:21:01.779906 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.779771 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 14:16:01 +0000 UTC" deadline="2028-01-31 07:49:24.087200903 +0000 UTC" Jul 13 14:21:01.779906 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.779848 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13601h28m22.307357127s" Jul 13 14:21:01.861826 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.861791 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:01.861996 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:01.861960 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:01.869387 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.869356 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbm2b" event={"ID":"a0d488fc-58f7-409e-ba15-7f7980cbd394","Type":"ContainerStarted","Data":"fb4c801a6f5a22dfd19ed0ac76827c62f0572538e5838af7fee29ee56fd298f1"} Jul 13 14:21:01.870804 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.870756 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" event={"ID":"d5ab8d67-8d30-4d28-be21-8210bedfea5b","Type":"ContainerStarted","Data":"5abecacfd230866e7a38d41c318870588290fbf93515b8c657647608961c03f8"} Jul 13 14:21:01.872159 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.872110 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" event={"ID":"fcd601a1-5d41-4039-bdb4-219a3e4aef87","Type":"ContainerStarted","Data":"82c2ff098a8101df05062888dfb69cf0c27c1cb8abbd80c21a04944778e33e83"} Jul 13 14:21:01.873418 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.873389 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hfmt7" event={"ID":"11a8e255-d719-4783-be00-d89aca5244c5","Type":"ContainerStarted","Data":"852de9a8acec450a733f5ae000ea57599d54bcd19fc66a6ef8ba8ba988ba5774"} Jul 13 14:21:01.875653 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.875624 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xcv69" event={"ID":"91eed05b-8ccb-4806-ac18-2ced768c9780","Type":"ContainerStarted","Data":"020b4532afd7de4d66f4a3fc5fc00efc1b7055fb836a5801652a8c8886df7d86"} Jul 13 14:21:01.877107 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.877087 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lvs6p" event={"ID":"58d081b3-2720-409f-8b1c-a82bc7238691","Type":"ContainerStarted","Data":"bb123097521692de8c9804bff31477ddd524162d91e49ce23386a39f639fab02"} Jul 13 14:21:01.878383 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.878359 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bmrdc" event={"ID":"abb92db6-649f-49f1-8ec1-5f41c88f9cfb","Type":"ContainerStarted","Data":"fb26de6bed6c6931f15fc09b582826bf510dcabc53cbae402b5a25f32124f667"} Jul 13 14:21:01.879771 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.879722 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" event={"ID":"663d50ff-7b28-4a5e-a211-6761789a969c","Type":"ContainerStarted","Data":"e33954852a439e0d43aa9a3f27c56472d59cf980218b48ea2192a1435e5be402"} Jul 13 14:21:01.880968 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:01.880935 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-76bmg" event={"ID":"5962361d-1783-4050-8470-a1b9518399bb","Type":"ContainerStarted","Data":"61a960c1654abb526b3f4d55800ac63092d577e0c7bab21339b5a50c53f303d2"} Jul 13 14:21:01.913344 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:01.913313 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6883fe87cd18b4634205e5c8665ef14.slice/crio-0b765ca563a7c7ba4c05eebbd34d4e3be7cd306a2cc0b0f49fdd9de8fa3f78ef WatchSource:0}: Error finding container 0b765ca563a7c7ba4c05eebbd34d4e3be7cd306a2cc0b0f49fdd9de8fa3f78ef: Status 404 returned error can't find the container with id 0b765ca563a7c7ba4c05eebbd34d4e3be7cd306a2cc0b0f49fdd9de8fa3f78ef Jul 13 14:21:02.356452 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:02.356410 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs\") pod \"network-metrics-daemon-cj24h\" (UID: \"cde58de7-b1b7-41c5-83c1-eaed967537b2\") " pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:02.356666 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:02.356511 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:02.356666 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:02.356637 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:21:02.356805 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:02.356703 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret podName:1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b nodeName:}" failed. No retries permitted until 2026-07-13 14:21:04.356684141 +0000 UTC m=+6.041104429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret") pod "global-pull-secret-syncer-f9wtx" (UID: "1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:21:02.357176 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:02.357154 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:21:02.357264 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:02.357217 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs podName:cde58de7-b1b7-41c5-83c1-eaed967537b2 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:04.357200363 +0000 UTC m=+6.041620652 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs") pod "network-metrics-daemon-cj24h" (UID: "cde58de7-b1b7-41c5-83c1-eaed967537b2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:21:02.457198 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:02.457158 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkkb\" (UniqueName: \"kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb\") pod \"network-check-target-sl54c\" (UID: \"4629c8d6-afb7-43da-b676-9d6418414491\") " pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:02.457378 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:02.457352 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:21:02.457378 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:02.457373 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:21:02.457501 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:02.457387 2566 projected.go:196] Error preparing data for projected volume kube-api-access-jgkkb for pod openshift-network-diagnostics/network-check-target-sl54c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:21:02.457501 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:02.457450 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb podName:4629c8d6-afb7-43da-b676-9d6418414491 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:04.457428566 +0000 UTC m=+6.141848874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgkkb" (UniqueName: "kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb") pod "network-check-target-sl54c" (UID: "4629c8d6-afb7-43da-b676-9d6418414491") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:21:02.780550 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:02.780452 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 14:16:01 +0000 UTC" deadline="2028-01-31 21:43:04.420363825 +0000 UTC" Jul 13 14:21:02.780550 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:02.780495 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13615h22m1.639873218s" Jul 13 14:21:02.873749 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:02.873061 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:02.873749 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:02.873200 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:02.873749 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:02.873610 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:02.873749 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:02.873698 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:02.898767 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:02.898684 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal" event={"ID":"c6883fe87cd18b4634205e5c8665ef14","Type":"ContainerStarted","Data":"0b765ca563a7c7ba4c05eebbd34d4e3be7cd306a2cc0b0f49fdd9de8fa3f78ef"} Jul 13 14:21:02.903160 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:02.902439 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.ec2.internal" event={"ID":"daa422171c2cfe1891a45971f28fd4ef","Type":"ContainerStarted","Data":"9462258d0a80812805817bd8274d4a3de4e8ff86b1a47ebcf7c7618152f09cc8"} Jul 13 14:21:03.861714 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:03.861671 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:03.862253 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:03.861854 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:04.388988 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:04.388944 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:04.389187 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:04.389016 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs\") pod \"network-metrics-daemon-cj24h\" (UID: \"cde58de7-b1b7-41c5-83c1-eaed967537b2\") " pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:04.389187 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:04.389141 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:21:04.389323 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:04.389206 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs podName:cde58de7-b1b7-41c5-83c1-eaed967537b2 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:08.389184866 +0000 UTC m=+10.073605153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs") pod "network-metrics-daemon-cj24h" (UID: "cde58de7-b1b7-41c5-83c1-eaed967537b2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:21:04.389824 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:04.389641 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:21:04.389824 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:04.389705 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret podName:1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b nodeName:}" failed. No retries permitted until 2026-07-13 14:21:08.389687955 +0000 UTC m=+10.074108259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret") pod "global-pull-secret-syncer-f9wtx" (UID: "1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:21:04.490141 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:04.490103 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkkb\" (UniqueName: \"kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb\") pod \"network-check-target-sl54c\" (UID: \"4629c8d6-afb7-43da-b676-9d6418414491\") " pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:04.490352 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:04.490320 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:21:04.490352 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:04.490345 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:21:04.490489 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:04.490357 2566 projected.go:196] Error preparing data for projected volume kube-api-access-jgkkb for pod openshift-network-diagnostics/network-check-target-sl54c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:21:04.490489 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:04.490416 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb podName:4629c8d6-afb7-43da-b676-9d6418414491 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:08.490397313 +0000 UTC m=+10.174817599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgkkb" (UniqueName: "kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb") pod "network-check-target-sl54c" (UID: "4629c8d6-afb7-43da-b676-9d6418414491") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:21:04.866469 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:04.866351 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:04.866469 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:04.866378 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:04.866952 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:04.866462 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:04.866952 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:04.866917 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:05.861932 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:05.861902 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:05.862119 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:05.862039 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:06.861835 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:06.861584 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:06.862271 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:06.861589 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:06.862271 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:06.861947 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:06.862271 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:06.862023 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:07.862512 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:07.862014 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:07.862512 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:07.862161 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:08.427046 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:08.426425 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:08.427046 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:08.426487 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs\") pod \"network-metrics-daemon-cj24h\" (UID: \"cde58de7-b1b7-41c5-83c1-eaed967537b2\") " pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:08.427046 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:08.426573 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:21:08.427046 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:08.426599 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:21:08.427046 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:08.426661 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs podName:cde58de7-b1b7-41c5-83c1-eaed967537b2 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:16.42664268 +0000 UTC m=+18.111062967 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs") pod "network-metrics-daemon-cj24h" (UID: "cde58de7-b1b7-41c5-83c1-eaed967537b2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:21:08.427046 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:08.426681 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret podName:1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b nodeName:}" failed. No retries permitted until 2026-07-13 14:21:16.426669894 +0000 UTC m=+18.111090183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret") pod "global-pull-secret-syncer-f9wtx" (UID: "1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:21:08.527349 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:08.527315 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkkb\" (UniqueName: \"kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb\") pod \"network-check-target-sl54c\" (UID: \"4629c8d6-afb7-43da-b676-9d6418414491\") " pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:08.527555 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:08.527532 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:21:08.527623 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:08.527563 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:21:08.527623 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:08.527578 2566 projected.go:196] Error preparing data for projected volume kube-api-access-jgkkb for pod openshift-network-diagnostics/network-check-target-sl54c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:21:08.527773 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:08.527641 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb podName:4629c8d6-afb7-43da-b676-9d6418414491 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:16.527620771 +0000 UTC m=+18.212041070 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgkkb" (UniqueName: "kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb") pod "network-check-target-sl54c" (UID: "4629c8d6-afb7-43da-b676-9d6418414491") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:21:08.862623 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:08.862542 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:08.863100 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:08.862655 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:08.863100 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:08.863035 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:08.863217 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:08.863127 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:09.861471 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:09.861439 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:09.861667 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:09.861583 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:10.861748 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:10.861649 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:10.862205 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:10.861767 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:10.862205 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:10.861836 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:10.862205 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:10.861909 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:11.862040 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:11.862002 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:11.862431 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:11.862159 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:12.862001 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:12.861966 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:12.862202 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:12.862093 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:12.862202 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:12.862153 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:12.862590 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:12.862268 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:13.862138 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:13.862094 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:13.862312 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:13.862243 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:14.861733 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:14.861664 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:14.861921 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:14.861800 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:14.861921 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:14.861837 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:14.862038 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:14.861928 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:15.861773 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:15.861737 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:15.862291 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:15.861885 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:16.483570 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:16.483534 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs\") pod \"network-metrics-daemon-cj24h\" (UID: \"cde58de7-b1b7-41c5-83c1-eaed967537b2\") " pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:16.483803 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:16.483628 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:16.483803 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:16.483737 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:21:16.483908 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:16.483816 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:21:16.483908 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:16.483830 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs podName:cde58de7-b1b7-41c5-83c1-eaed967537b2 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:32.483813126 +0000 UTC m=+34.168233409 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs") pod "network-metrics-daemon-cj24h" (UID: "cde58de7-b1b7-41c5-83c1-eaed967537b2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:21:16.483908 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:16.483867 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret podName:1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b nodeName:}" failed. No retries permitted until 2026-07-13 14:21:32.483850997 +0000 UTC m=+34.168271285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret") pod "global-pull-secret-syncer-f9wtx" (UID: "1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:21:16.584628 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:16.584588 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkkb\" (UniqueName: \"kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb\") pod \"network-check-target-sl54c\" (UID: \"4629c8d6-afb7-43da-b676-9d6418414491\") " pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:16.584819 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:16.584800 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:21:16.584899 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:16.584825 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:21:16.584899 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:16.584839 2566 projected.go:196] Error preparing data for projected volume kube-api-access-jgkkb for pod openshift-network-diagnostics/network-check-target-sl54c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:21:16.584995 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:16.584909 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb podName:4629c8d6-afb7-43da-b676-9d6418414491 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:32.584890664 +0000 UTC m=+34.269310961 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgkkb" (UniqueName: "kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb") pod "network-check-target-sl54c" (UID: "4629c8d6-afb7-43da-b676-9d6418414491") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:21:16.862154 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:16.862074 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:16.862539 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:16.862202 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:16.862539 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:16.862258 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:16.862539 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:16.862364 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:17.862144 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:17.862114 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:17.862297 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:17.862249 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:18.862340 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:18.862308 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:18.862908 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:18.862417 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:18.862908 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:18.862472 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:18.862908 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:18.862532 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:19.861581 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:19.861542 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:19.861753 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:19.861675 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:20.861876 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.861690 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:20.862664 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.861700 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:20.862664 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:20.861970 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:20.862664 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:20.862030 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:20.945699 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.945503 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xcv69" event={"ID":"91eed05b-8ccb-4806-ac18-2ced768c9780","Type":"ContainerStarted","Data":"d75605db8d4f978e23da5aa8abb47a5fcc43bcbfb4ce5fa455ae813de42c2733"} Jul 13 14:21:20.947109 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.947084 2566 generic.go:356] "Generic (PLEG): container finished" podID="c6883fe87cd18b4634205e5c8665ef14" containerID="e28397d84d7dff00f8aac17438a090804ebb6076b463b11a8f42376015938653" exitCode=0 Jul 13 14:21:20.947198 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.947156 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal" event={"ID":"c6883fe87cd18b4634205e5c8665ef14","Type":"ContainerDied","Data":"e28397d84d7dff00f8aac17438a090804ebb6076b463b11a8f42376015938653"} Jul 13 14:21:20.948600 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.948564 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bmrdc" event={"ID":"abb92db6-649f-49f1-8ec1-5f41c88f9cfb","Type":"ContainerStarted","Data":"b3eb35f8aa10aad218825f7ed98c73e1d54a9cb9838e23bc91428f21fdb0c52b"} Jul 13 14:21:20.950012 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.949989 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" event={"ID":"663d50ff-7b28-4a5e-a211-6761789a969c","Type":"ContainerStarted","Data":"006c772082ecb106e72b6ac94414a59ed4d36524a0edc5971cc0af48880f0d71"} Jul 13 14:21:20.951175 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.951154 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-76bmg" event={"ID":"5962361d-1783-4050-8470-a1b9518399bb","Type":"ContainerStarted","Data":"746a9ad1c8ca815bde359a9ecd4033b154314bd4b2bbddf4393b1a1e49c12be7"} Jul 13 14:21:20.952493 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.952473 2566 generic.go:356] "Generic (PLEG): container finished" podID="a0d488fc-58f7-409e-ba15-7f7980cbd394" containerID="3931a9c91507d76590b2623b2d92237eeaebffa510b24c60200d9b426fd8e1b5" exitCode=0 Jul 13 14:21:20.952573 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.952532 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbm2b" event={"ID":"a0d488fc-58f7-409e-ba15-7f7980cbd394","Type":"ContainerDied","Data":"3931a9c91507d76590b2623b2d92237eeaebffa510b24c60200d9b426fd8e1b5"} Jul 13 14:21:20.954027 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.954010 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" event={"ID":"d5ab8d67-8d30-4d28-be21-8210bedfea5b","Type":"ContainerStarted","Data":"0a60666029fb1b84197300db2027b46ca37fa423b917b10306ada2b58839cb47"} Jul 13 14:21:20.955939 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.955922 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" event={"ID":"fcd601a1-5d41-4039-bdb4-219a3e4aef87","Type":"ContainerStarted","Data":"3dfdc1dd2f4cf8faa10199b5af9d809cb6c8a8387ee570bc6af7052bf3030b62"} Jul 13 14:21:20.956009 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.955984 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" event={"ID":"fcd601a1-5d41-4039-bdb4-219a3e4aef87","Type":"ContainerStarted","Data":"f0b89ee318503f7b2d5c7a649142bcec6c31fa5081874642952d017ac9466e59"} Jul 13 14:21:20.956044 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.956031 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" event={"ID":"fcd601a1-5d41-4039-bdb4-219a3e4aef87","Type":"ContainerStarted","Data":"76bc73ea89d1eb7c1e37ae6a7a05c2abd0f2a13bec38e2fc8efabcaec816f0fb"} Jul 13 14:21:20.957207 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.957191 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hfmt7" event={"ID":"11a8e255-d719-4783-be00-d89aca5244c5","Type":"ContainerStarted","Data":"a4429872459d7a6674d5d6f8ffd6d6a6b91b1bcca625d47d0ff7d3acbe4e12c5"} Jul 13 14:21:20.964731 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:20.964659 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-188.ec2.internal" podStartSLOduration=20.964642614 podStartE2EDuration="20.964642614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:02.927268543 +0000 UTC m=+4.611688854" watchObservedRunningTime="2026-07-13 14:21:20.964642614 +0000 UTC m=+22.649062917" Jul 13 14:21:21.001760 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:21.001697 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xcv69" podStartSLOduration=2.6109415780000003 podStartE2EDuration="22.001680491s" totalImagesPullingTime="19.390738913s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:20.965202323 +0000 UTC m=+22.649622642" watchObservedRunningTime="2026-07-13 14:21:21.001680491 +0000 UTC m=+22.686100796" Jul 13 14:21:21.046307 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:21.046263 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r9qr6" podStartSLOduration=2.65048704 podStartE2EDuration="22.04624743s" totalImagesPullingTime="19.39576039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:21.045833309 +0000 UTC m=+22.730253615" watchObservedRunningTime="2026-07-13 14:21:21.04624743 +0000 UTC m=+22.730667735" Jul 13 14:21:21.063714 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:21.063672 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bmrdc" podStartSLOduration=10.756010038 podStartE2EDuration="22.063653583s" totalImagesPullingTime="11.307643545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:21.063105791 +0000 UTC m=+22.747526096" watchObservedRunningTime="2026-07-13 14:21:21.063653583 +0000 UTC m=+22.748073890" Jul 13 14:21:21.082067 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:21.082028 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-76bmg" podStartSLOduration=2.696201609 podStartE2EDuration="22.082014501s" totalImagesPullingTime="19.385812892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:21.081751292 +0000 UTC m=+22.766171609" watchObservedRunningTime="2026-07-13 14:21:21.082014501 +0000 UTC m=+22.766434807" Jul 13 14:21:21.101015 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:21.100969 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hfmt7" podStartSLOduration=2.677683886 podStartE2EDuration="22.100954418s" totalImagesPullingTime="19.423270532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:21.100358877 +0000 UTC m=+22.784779182" watchObservedRunningTime="2026-07-13 14:21:21.100954418 +0000 UTC m=+22.785374724" Jul 13 14:21:21.861245 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:21.861212 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:21.861473 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:21.861319 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:21.962450 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:21.962417 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" event={"ID":"fcd601a1-5d41-4039-bdb4-219a3e4aef87","Type":"ContainerStarted","Data":"3275923be7e69221f1bfba049f955744456f428d5d2097b7b65b84cf25b96da9"} Jul 13 14:21:21.963136 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:21.962485 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" event={"ID":"fcd601a1-5d41-4039-bdb4-219a3e4aef87","Type":"ContainerStarted","Data":"7ed87e60b336cf799e08806739dd687855c815ef50d69d308a868977eb28d6ee"} Jul 13 14:21:21.963136 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:21.962525 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" event={"ID":"fcd601a1-5d41-4039-bdb4-219a3e4aef87","Type":"ContainerStarted","Data":"1ec4228a9ad438d25c9d16dd88e0247b1ef4007e008e806d4ef26db2eb363d3b"} Jul 13 14:21:21.964154 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:21.964123 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal" event={"ID":"c6883fe87cd18b4634205e5c8665ef14","Type":"ContainerStarted","Data":"e96bde0a88928012438cf322415b869e29248fd054cea18b61f72d19b8d03e35"} Jul 13 14:21:21.965500 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:21.965476 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lvs6p" event={"ID":"58d081b3-2720-409f-8b1c-a82bc7238691","Type":"ContainerStarted","Data":"0b65a63e2a196be352c40b403788d03ba6596b783eadc344291071e8c1d60a38"} Jul 13 14:21:22.008682 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:22.008589 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-188.ec2.internal" podStartSLOduration=22.00857314 podStartE2EDuration="22.00857314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:21.988391804 +0000 UTC m=+23.672812110" watchObservedRunningTime="2026-07-13 14:21:22.00857314 +0000 UTC m=+23.692993446" Jul 13 14:21:22.008938 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:22.008915 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lvs6p" podStartSLOduration=3.679064361 podStartE2EDuration="23.008909429s" totalImagesPullingTime="19.329845068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:22.008689014 +0000 UTC m=+23.693109321" watchObservedRunningTime="2026-07-13 14:21:22.008909429 +0000 UTC m=+23.693329714" Jul 13 14:21:22.237447 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:22.237416 2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 13 14:21:22.690761 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:22.690721 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bmrdc" Jul 13 14:21:22.691500 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:22.691475 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bmrdc" Jul 13 14:21:22.799504 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:22.799394 2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T14:21:22.237441678Z","UUID":"d7a55c7e-678f-41ce-9446-6b1489382af9","Handler":null,"Name":"","Endpoint":""} Jul 13 14:21:22.801158 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:22.801134 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 13 14:21:22.801288 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:22.801178 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 13 14:21:22.861849 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:22.861775 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:22.862037 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:22.861775 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:22.862111 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:22.861955 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:22.862111 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:22.862066 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:22.969036 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:22.968961 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" event={"ID":"d5ab8d67-8d30-4d28-be21-8210bedfea5b","Type":"ContainerStarted","Data":"addf51f5a0e13fe02d7e6b30928948cc3b4f720efcc277f5f6a2c59c21bee990"} Jul 13 14:21:22.969598 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:22.969363 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bmrdc" Jul 13 14:21:22.969835 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:22.969819 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bmrdc" Jul 13 14:21:23.861548 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:23.861289 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:23.861716 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:23.861597 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:23.972856 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:23.972821 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" event={"ID":"d5ab8d67-8d30-4d28-be21-8210bedfea5b","Type":"ContainerStarted","Data":"94ed32c0a12d2257390dccb78e0df3e55ca55357baa7604dea244a3db9ac04dc"} Jul 13 14:21:23.975902 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:23.975878 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" event={"ID":"fcd601a1-5d41-4039-bdb4-219a3e4aef87","Type":"ContainerStarted","Data":"fc33f77995816facdfdaaa3aaa7cc2a2e1163b7ef4c8def29a6f1c37c6c9e435"} Jul 13 14:21:24.862172 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:24.862136 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:24.862379 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:24.862267 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:24.862379 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:24.862292 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:24.862502 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:24.862404 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:25.099222 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:25.099181 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:25.228474 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:25.228399 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:25.738215 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:25.738170 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:25.861872 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:25.861838 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:25.862027 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:25.861961 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:25.869778 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:25.869732 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:25.972165 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:25.972136 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:26.111719 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:26.111638 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:26.202753 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:26.202729 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:26.295841 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:26.295752 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:26.862061 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:26.861847 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:26.862247 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:26.861860 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:26.862247 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:26.862084 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:26.862247 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:26.862175 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:26.983424 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:26.983398 2566 generic.go:356] "Generic (PLEG): container finished" podID="a0d488fc-58f7-409e-ba15-7f7980cbd394" containerID="24f106a6826ba5374775bcd5ed4a57e4969622f6f182db1665b82d82a813974f" exitCode=0 Jul 13 14:21:26.983608 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:26.983480 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbm2b" event={"ID":"a0d488fc-58f7-409e-ba15-7f7980cbd394","Type":"ContainerDied","Data":"24f106a6826ba5374775bcd5ed4a57e4969622f6f182db1665b82d82a813974f"} Jul 13 14:21:26.987073 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:26.987048 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" event={"ID":"fcd601a1-5d41-4039-bdb4-219a3e4aef87","Type":"ContainerStarted","Data":"bb4694daac82e8df34f05d281596f805c6b4b3e7b346d1afc8f5fc3bea5da9bf"} Jul 13 14:21:26.987295 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:26.987284 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:26.987351 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:26.987331 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:27.002547 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:27.002529 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:27.012979 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:27.012937 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f6b84" podStartSLOduration=6.172590488 podStartE2EDuration="28.012925006s" totalImagesPullingTime="21.840334518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:20:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:23.997256698 +0000 UTC m=+25.681677003" watchObservedRunningTime="2026-07-13 14:21:27.012925006 +0000 UTC m=+28.697345312" Jul 13 14:21:27.045435 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:27.045393 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" podStartSLOduration=8.624589257 podStartE2EDuration="28.045381369s" totalImagesPullingTime="19.420792112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:27.045101864 +0000 UTC m=+28.729522170" watchObservedRunningTime="2026-07-13 14:21:27.045381369 +0000 UTC m=+28.729801675" Jul 13 14:21:27.861480 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:27.861448 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:27.862167 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:27.861578 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:27.989697 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:27.989671 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:28.003559 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:28.003534 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:21:28.165723 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:28.165648 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f9wtx"] Jul 13 14:21:28.165950 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:28.165775 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:28.165950 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:28.165901 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:28.168397 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:28.168374 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sl54c"] Jul 13 14:21:28.168535 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:28.168466 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:28.168596 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:28.168549 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:28.169055 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:28.169035 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cj24h"] Jul 13 14:21:28.169163 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:28.169118 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:28.169249 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:28.169229 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:29.367455 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:29.367382 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:29.436941 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:29.436910 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:29.861430 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:29.861272 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:29.861560 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:29.861290 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:29.861560 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:29.861519 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:29.861677 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:29.861597 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:29.861677 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:29.861289 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:29.861769 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:29.861703 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:29.995667 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:29.995638 2566 generic.go:356] "Generic (PLEG): container finished" podID="a0d488fc-58f7-409e-ba15-7f7980cbd394" containerID="8f61223a0507cab07d32fc55075dfde68b11e6841f3f6ba8a0a28a40b7be3788" exitCode=0 Jul 13 14:21:29.995832 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:29.995727 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbm2b" event={"ID":"a0d488fc-58f7-409e-ba15-7f7980cbd394","Type":"ContainerDied","Data":"8f61223a0507cab07d32fc55075dfde68b11e6841f3f6ba8a0a28a40b7be3788"} Jul 13 14:21:31.862232 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:31.862197 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:31.862585 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:31.862237 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:31.862585 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:31.862246 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:31.862585 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:31.862338 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cj24h" podUID="cde58de7-b1b7-41c5-83c1-eaed967537b2" Jul 13 14:21:31.862585 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:31.862421 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sl54c" podUID="4629c8d6-afb7-43da-b676-9d6418414491" Jul 13 14:21:31.862585 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:31.862482 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f9wtx" podUID="1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b" Jul 13 14:21:32.497823 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:32.497778 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs\") pod \"network-metrics-daemon-cj24h\" (UID: \"cde58de7-b1b7-41c5-83c1-eaed967537b2\") " pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:32.497934 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:32.497918 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:21:32.497975 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:32.497944 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:32.498008 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:32.497972 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs podName:cde58de7-b1b7-41c5-83c1-eaed967537b2 nodeName:}" failed. No retries permitted until 2026-07-13 14:22:04.497957785 +0000 UTC m=+66.182378068 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs") pod "network-metrics-daemon-cj24h" (UID: "cde58de7-b1b7-41c5-83c1-eaed967537b2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 14:21:32.498051 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:32.498029 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 14:21:32.498080 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:32.498076 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret podName:1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b nodeName:}" failed. No retries permitted until 2026-07-13 14:22:04.498063357 +0000 UTC m=+66.182483640 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret") pod "global-pull-secret-syncer-f9wtx" (UID: "1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b") : object "kube-system"/"original-pull-secret" not registered Jul 13 14:21:32.598403 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:32.598340 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkkb\" (UniqueName: \"kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb\") pod \"network-check-target-sl54c\" (UID: \"4629c8d6-afb7-43da-b676-9d6418414491\") " pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:32.598510 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:32.598463 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 14:21:32.598510 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:32.598475 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 14:21:32.598510 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:32.598483 2566 projected.go:196] Error preparing data for projected volume kube-api-access-jgkkb for pod openshift-network-diagnostics/network-check-target-sl54c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:21:32.598637 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:32.598523 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb podName:4629c8d6-afb7-43da-b676-9d6418414491 nodeName:}" failed. No retries permitted until 2026-07-13 14:22:04.59851148 +0000 UTC m=+66.282931764 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jgkkb" (UniqueName: "kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb") pod "network-check-target-sl54c" (UID: "4629c8d6-afb7-43da-b676-9d6418414491") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 14:21:33.004443 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.004414 2566 generic.go:356] "Generic (PLEG): container finished" podID="a0d488fc-58f7-409e-ba15-7f7980cbd394" containerID="d690240e5a2a0365b9f8de50d9fdd64483cbfc2db3c73e64e670118641792d9a" exitCode=0 Jul 13 14:21:33.004777 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.004468 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbm2b" event={"ID":"a0d488fc-58f7-409e-ba15-7f7980cbd394","Type":"ContainerDied","Data":"d690240e5a2a0365b9f8de50d9fdd64483cbfc2db3c73e64e670118641792d9a"} Jul 13 14:21:33.194462 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.194433 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-188.ec2.internal" event="NodeReady" Jul 13 14:21:33.194601 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.194554 2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 13 14:21:33.249215 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.249146 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5647674757-gwgns"] Jul 13 14:21:33.251156 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.251140 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.259361 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.259307 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 13 14:21:33.259463 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.259382 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 13 14:21:33.259623 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.259610 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fdc44\"" Jul 13 14:21:33.263853 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.263838 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 13 14:21:33.267608 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.267589 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 13 14:21:33.271076 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.271058 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bhvr5"] Jul 13 14:21:33.272859 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.272845 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:33.274644 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.274622 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5647674757-gwgns"] Jul 13 14:21:33.275932 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.275910 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 13 14:21:33.276015 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.275952 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6lrx7\"" Jul 13 14:21:33.276293 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.276269 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 13 14:21:33.291821 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.291777 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6dmbf"] Jul 13 14:21:33.294827 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.294804 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6dmbf" Jul 13 14:21:33.294939 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.294813 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bhvr5"] Jul 13 14:21:33.297851 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.297831 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 13 14:21:33.298530 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.298505 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 13 14:21:33.298702 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.298685 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9jbjr\"" Jul 13 14:21:33.298769 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.298738 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 13 14:21:33.304314 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.304294 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6dmbf"] Jul 13 14:21:33.403836 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.403808 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-certificates\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.403836 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.403838 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:33.404040 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.403856 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t9zm\" (UniqueName: \"kubernetes.io/projected/ea289c9c-fdbc-4830-ad88-6d6fbf43834d-kube-api-access-7t9zm\") pod \"ingress-canary-6dmbf\" (UID: \"ea289c9c-fdbc-4830-ad88-6d6fbf43834d\") " pod="openshift-ingress-canary/ingress-canary-6dmbf" Jul 13 14:21:33.404040 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.403875 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.404040 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.403891 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9f13eee-31c5-40f5-a420-c1214e371eb2-installation-pull-secrets\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.404040 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.403970 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm7fz\" (UniqueName: \"kubernetes.io/projected/58085733-1f46-4db6-9ae7-1eae8fec5bab-kube-api-access-fm7fz\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:33.404040 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.403997 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea289c9c-fdbc-4830-ad88-6d6fbf43834d-cert\") pod \"ingress-canary-6dmbf\" (UID: \"ea289c9c-fdbc-4830-ad88-6d6fbf43834d\") " pod="openshift-ingress-canary/ingress-canary-6dmbf" Jul 13 14:21:33.404040 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.404015 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58085733-1f46-4db6-9ae7-1eae8fec5bab-tmp-dir\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:33.404242 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.404042 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqvm\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-kube-api-access-ztqvm\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.404242 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.404067 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9f13eee-31c5-40f5-a420-c1214e371eb2-trusted-ca\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.404242 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.404102 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58085733-1f46-4db6-9ae7-1eae8fec5bab-config-volume\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:33.404242 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.404138 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9f13eee-31c5-40f5-a420-c1214e371eb2-image-registry-private-configuration\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.404242 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.404167 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9f13eee-31c5-40f5-a420-c1214e371eb2-ca-trust-extracted\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.404242 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.404197 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-bound-sa-token\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.489437 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.489405 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:33.505493 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.505423 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-certificates\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.505493 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.505459 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:33.505493 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.505479 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t9zm\" (UniqueName: \"kubernetes.io/projected/ea289c9c-fdbc-4830-ad88-6d6fbf43834d-kube-api-access-7t9zm\") pod \"ingress-canary-6dmbf\" (UID: \"ea289c9c-fdbc-4830-ad88-6d6fbf43834d\") " pod="openshift-ingress-canary/ingress-canary-6dmbf" Jul 13 14:21:33.505677 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.505501 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.505677 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:33.505599 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 14:21:33.505677 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:33.505613 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5647674757-gwgns: secret "image-registry-tls" not found Jul 13 14:21:33.505677 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.505628 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9f13eee-31c5-40f5-a420-c1214e371eb2-installation-pull-secrets\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.505677 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:33.505672 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls podName:b9f13eee-31c5-40f5-a420-c1214e371eb2 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:34.005653187 +0000 UTC m=+35.690073473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls") pod "image-registry-5647674757-gwgns" (UID: "b9f13eee-31c5-40f5-a420-c1214e371eb2") : secret "image-registry-tls" not found Jul 13 14:21:33.505931 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:33.505600 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:21:33.505931 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.505722 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm7fz\" (UniqueName: \"kubernetes.io/projected/58085733-1f46-4db6-9ae7-1eae8fec5bab-kube-api-access-fm7fz\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:33.505931 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:33.505829 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls podName:58085733-1f46-4db6-9ae7-1eae8fec5bab nodeName:}" failed. No retries permitted until 2026-07-13 14:21:34.005813001 +0000 UTC m=+35.690233299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls") pod "dns-default-bhvr5" (UID: "58085733-1f46-4db6-9ae7-1eae8fec5bab") : secret "dns-default-metrics-tls" not found Jul 13 14:21:33.505931 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.505848 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea289c9c-fdbc-4830-ad88-6d6fbf43834d-cert\") pod \"ingress-canary-6dmbf\" (UID: \"ea289c9c-fdbc-4830-ad88-6d6fbf43834d\") " pod="openshift-ingress-canary/ingress-canary-6dmbf" Jul 13 14:21:33.505931 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.505877 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58085733-1f46-4db6-9ae7-1eae8fec5bab-tmp-dir\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:33.505931 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.505899 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqvm\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-kube-api-access-ztqvm\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.505931 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.505921 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9f13eee-31c5-40f5-a420-c1214e371eb2-trusted-ca\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.506256 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.505944 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58085733-1f46-4db6-9ae7-1eae8fec5bab-config-volume\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:33.506256 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.505970 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9f13eee-31c5-40f5-a420-c1214e371eb2-image-registry-private-configuration\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.506256 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.505999 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9f13eee-31c5-40f5-a420-c1214e371eb2-ca-trust-extracted\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.506256 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.506032 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-bound-sa-token\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.506256 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.506158 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-certificates\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.506572 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.506547 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9f13eee-31c5-40f5-a420-c1214e371eb2-ca-trust-extracted\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.506644 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.506550 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58085733-1f46-4db6-9ae7-1eae8fec5bab-config-volume\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:33.506734 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.506713 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58085733-1f46-4db6-9ae7-1eae8fec5bab-tmp-dir\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:33.507108 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.507067 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9f13eee-31c5-40f5-a420-c1214e371eb2-trusted-ca\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.510384 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.510366 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9f13eee-31c5-40f5-a420-c1214e371eb2-image-registry-private-configuration\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.510486 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.510408 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea289c9c-fdbc-4830-ad88-6d6fbf43834d-cert\") pod \"ingress-canary-6dmbf\" (UID: \"ea289c9c-fdbc-4830-ad88-6d6fbf43834d\") " pod="openshift-ingress-canary/ingress-canary-6dmbf" Jul 13 14:21:33.510486 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.510416 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9f13eee-31c5-40f5-a420-c1214e371eb2-installation-pull-secrets\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.515656 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.515633 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-bound-sa-token\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.516613 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.516594 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t9zm\" (UniqueName: \"kubernetes.io/projected/ea289c9c-fdbc-4830-ad88-6d6fbf43834d-kube-api-access-7t9zm\") pod \"ingress-canary-6dmbf\" (UID: \"ea289c9c-fdbc-4830-ad88-6d6fbf43834d\") " pod="openshift-ingress-canary/ingress-canary-6dmbf" Jul 13 14:21:33.516727 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.516660 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqvm\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-kube-api-access-ztqvm\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:33.516898 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.516878 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm7fz\" (UniqueName: \"kubernetes.io/projected/58085733-1f46-4db6-9ae7-1eae8fec5bab-kube-api-access-fm7fz\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:33.590333 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.590294 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:33.603328 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.603305 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6dmbf" Jul 13 14:21:33.748897 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.748649 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6dmbf"] Jul 13 14:21:33.754329 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:33.754301 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea289c9c_fdbc_4830_ad88_6d6fbf43834d.slice/crio-bb4befdbfa44069b80286443567055e27eb5ff6ee5d12b7c93ef9a90c882ef56 WatchSource:0}: Error finding container bb4befdbfa44069b80286443567055e27eb5ff6ee5d12b7c93ef9a90c882ef56: Status 404 returned error can't find the container with id bb4befdbfa44069b80286443567055e27eb5ff6ee5d12b7c93ef9a90c882ef56 Jul 13 14:21:33.862149 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.862122 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:21:33.862324 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.862154 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:21:33.862324 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.862119 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:21:33.865685 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.865661 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 14:21:33.865831 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.865713 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 14:21:33.865831 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.865664 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 14:21:33.866005 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.865988 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 14:21:33.866355 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.866338 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5z6w6\"" Jul 13 14:21:33.866711 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:33.866689 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hslvn\"" Jul 13 14:21:34.007702 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:34.007616 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6dmbf" event={"ID":"ea289c9c-fdbc-4830-ad88-6d6fbf43834d","Type":"ContainerStarted","Data":"bb4befdbfa44069b80286443567055e27eb5ff6ee5d12b7c93ef9a90c882ef56"} Jul 13 14:21:34.011003 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:34.010978 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:34.011097 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:34.011016 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:34.011144 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:34.011108 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:21:34.011185 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:34.011159 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls podName:58085733-1f46-4db6-9ae7-1eae8fec5bab nodeName:}" failed. No retries permitted until 2026-07-13 14:21:35.011144498 +0000 UTC m=+36.695564781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls") pod "dns-default-bhvr5" (UID: "58085733-1f46-4db6-9ae7-1eae8fec5bab") : secret "dns-default-metrics-tls" not found Jul 13 14:21:34.011185 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:34.011109 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 14:21:34.011185 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:34.011183 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5647674757-gwgns: secret "image-registry-tls" not found Jul 13 14:21:34.011336 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:34.011229 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls podName:b9f13eee-31c5-40f5-a420-c1214e371eb2 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:35.011215255 +0000 UTC m=+36.695635556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls") pod "image-registry-5647674757-gwgns" (UID: "b9f13eee-31c5-40f5-a420-c1214e371eb2") : secret "image-registry-tls" not found Jul 13 14:21:35.019568 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.019532 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:35.020019 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.019581 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:35.020019 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:35.019702 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 14:21:35.020019 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:35.019717 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5647674757-gwgns: secret "image-registry-tls" not found Jul 13 14:21:35.020019 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:35.019702 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:21:35.020019 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:35.019780 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls podName:b9f13eee-31c5-40f5-a420-c1214e371eb2 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:37.019762491 +0000 UTC m=+38.704182779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls") pod "image-registry-5647674757-gwgns" (UID: "b9f13eee-31c5-40f5-a420-c1214e371eb2") : secret "image-registry-tls" not found Jul 13 14:21:35.020019 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:35.019815 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls podName:58085733-1f46-4db6-9ae7-1eae8fec5bab nodeName:}" failed. No retries permitted until 2026-07-13 14:21:37.019805333 +0000 UTC m=+38.704225619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls") pod "dns-default-bhvr5" (UID: "58085733-1f46-4db6-9ae7-1eae8fec5bab") : secret "dns-default-metrics-tls" not found Jul 13 14:21:35.306411 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.306332 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:35.484899 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.484869 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:35.607169 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.607089 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:35.703380 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.703348 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:35.818257 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.818221 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:35.886632 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.886555 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-z78ln"] Jul 13 14:21:35.889354 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.889331 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:35.892252 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.892225 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 13 14:21:35.892378 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.892225 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mcjrd\"" Jul 13 14:21:35.892378 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.892250 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 13 14:21:35.893192 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.893175 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 13 14:21:35.893294 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.893201 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 13 14:21:35.928429 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.928401 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-crio-socket\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:35.928565 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.928440 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-data-volume\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:35.928565 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.928464 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:35.928565 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.928504 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz56t\" (UniqueName: \"kubernetes.io/projected/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-api-access-pz56t\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:35.928720 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.928569 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:35.946037 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.946008 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:35.956459 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:35.956422 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z78ln"] Jul 13 14:21:36.029873 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.029837 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-crio-socket\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:36.030403 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.029883 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-data-volume\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:36.030403 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.029905 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:36.030403 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.029926 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz56t\" (UniqueName: \"kubernetes.io/projected/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-api-access-pz56t\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:36.030403 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.029963 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:36.030403 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:36.030043 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 14:21:36.030403 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:36.030069 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:21:36.030403 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.030069 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-crio-socket\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:36.030403 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:36.030117 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-rbac-proxy-cm podName:bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe nodeName:}" failed. No retries permitted until 2026-07-13 14:21:36.530098391 +0000 UTC m=+38.214518675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z78ln" (UID: "bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe") : configmap "kube-rbac-proxy" not found Jul 13 14:21:36.030403 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:36.030134 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls podName:bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe nodeName:}" failed. No retries permitted until 2026-07-13 14:21:36.530126274 +0000 UTC m=+38.214546564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z78ln" (UID: "bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe") : secret "insights-runtime-extractor-tls" not found Jul 13 14:21:36.030403 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.030322 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-data-volume\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:36.048823 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.048778 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz56t\" (UniqueName: \"kubernetes.io/projected/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-api-access-pz56t\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:36.281465 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.281381 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:36.318681 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.318635 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bj582"] Jul 13 14:21:36.321055 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.321029 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bj582" Jul 13 14:21:36.323816 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.323764 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-f2824\"" Jul 13 14:21:36.324254 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.324231 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 13 14:21:36.324364 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.324273 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 13 14:21:36.332269 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.332246 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx8xh\" (UniqueName: \"kubernetes.io/projected/88a63d66-f350-425c-b173-98c532394025-kube-api-access-tx8xh\") pod \"migrator-7f59b54f6c-bj582\" (UID: \"88a63d66-f350-425c-b173-98c532394025\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bj582" Jul 13 14:21:36.336292 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.336271 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bj582"] Jul 13 14:21:36.382345 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.382306 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:36.433518 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.433486 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx8xh\" (UniqueName: \"kubernetes.io/projected/88a63d66-f350-425c-b173-98c532394025-kube-api-access-tx8xh\") pod \"migrator-7f59b54f6c-bj582\" (UID: \"88a63d66-f350-425c-b173-98c532394025\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bj582" Jul 13 14:21:36.448354 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.448332 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx8xh\" (UniqueName: \"kubernetes.io/projected/88a63d66-f350-425c-b173-98c532394025-kube-api-access-tx8xh\") pod \"migrator-7f59b54f6c-bj582\" (UID: \"88a63d66-f350-425c-b173-98c532394025\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bj582" Jul 13 14:21:36.474341 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.474313 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:36.534293 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.534207 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:36.534293 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.534266 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:36.534508 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:36.534332 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 14:21:36.534508 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:36.534402 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-rbac-proxy-cm podName:bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe nodeName:}" failed. No retries permitted until 2026-07-13 14:21:37.534385847 +0000 UTC m=+39.218806133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z78ln" (UID: "bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe") : configmap "kube-rbac-proxy" not found Jul 13 14:21:36.534508 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:36.534408 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:21:36.534508 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:36.534458 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls podName:bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe nodeName:}" failed. No retries permitted until 2026-07-13 14:21:37.534443338 +0000 UTC m=+39.218863637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z78ln" (UID: "bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe") : secret "insights-runtime-extractor-tls" not found Jul 13 14:21:36.631968 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.631935 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bj582" Jul 13 14:21:36.713233 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.713184 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:36.840045 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.839927 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:36.985236 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:36.985206 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:37.038544 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.038508 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:37.039147 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.038553 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:37.039147 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:37.038669 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:21:37.039147 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:37.038704 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 14:21:37.039147 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:37.038720 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5647674757-gwgns: secret "image-registry-tls" not found Jul 13 14:21:37.039147 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:37.038732 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls podName:58085733-1f46-4db6-9ae7-1eae8fec5bab nodeName:}" failed. No retries permitted until 2026-07-13 14:21:41.03871324 +0000 UTC m=+42.723133528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls") pod "dns-default-bhvr5" (UID: "58085733-1f46-4db6-9ae7-1eae8fec5bab") : secret "dns-default-metrics-tls" not found Jul 13 14:21:37.039147 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:37.038760 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls podName:b9f13eee-31c5-40f5-a420-c1214e371eb2 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:41.038748854 +0000 UTC m=+42.723169142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls") pod "image-registry-5647674757-gwgns" (UID: "b9f13eee-31c5-40f5-a420-c1214e371eb2") : secret "image-registry-tls" not found Jul 13 14:21:37.081652 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.081620 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:37.385768 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.385721 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc"] Jul 13 14:21:37.391712 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.391687 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" Jul 13 14:21:37.403214 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.403194 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 13 14:21:37.403759 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.403737 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 13 14:21:37.403889 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.403872 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 13 14:21:37.404900 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.404882 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 13 14:21:37.411492 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.411473 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc"] Jul 13 14:21:37.413292 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.413274 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh"] Jul 13 14:21:37.416185 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.416162 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.421685 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.421663 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 13 14:21:37.421813 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.421762 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 13 14:21:37.421813 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.421776 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 13 14:21:37.421984 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.421964 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 13 14:21:37.435876 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.435854 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh"] Jul 13 14:21:37.442244 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.442224 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d0bab13-bd2b-4494-9177-7709e5a27d3c-ca\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.442347 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.442256 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5760faea-8ad3-45fc-b57f-25337f6b027e-tmp\") pod \"klusterlet-addon-workmgr-797bb4bf7f-tbtsc\" (UID: \"5760faea-8ad3-45fc-b57f-25337f6b027e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" Jul 13 14:21:37.442347 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.442298 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5760faea-8ad3-45fc-b57f-25337f6b027e-klusterlet-config\") pod \"klusterlet-addon-workmgr-797bb4bf7f-tbtsc\" (UID: \"5760faea-8ad3-45fc-b57f-25337f6b027e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" Jul 13 14:21:37.442347 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.442330 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0bab13-bd2b-4494-9177-7709e5a27d3c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.442474 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.442415 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d0bab13-bd2b-4494-9177-7709e5a27d3c-hub\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.442474 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.442455 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkgp8\" (UniqueName: \"kubernetes.io/projected/5760faea-8ad3-45fc-b57f-25337f6b027e-kube-api-access-lkgp8\") pod \"klusterlet-addon-workmgr-797bb4bf7f-tbtsc\" (UID: \"5760faea-8ad3-45fc-b57f-25337f6b027e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" Jul 13 14:21:37.442546 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.442496 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d0bab13-bd2b-4494-9177-7709e5a27d3c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.442546 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.442522 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d0bab13-bd2b-4494-9177-7709e5a27d3c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.442633 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.442616 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75jvp\" (UniqueName: \"kubernetes.io/projected/2d0bab13-bd2b-4494-9177-7709e5a27d3c-kube-api-access-75jvp\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.543392 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.543358 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0bab13-bd2b-4494-9177-7709e5a27d3c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.543570 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.543423 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:37.543570 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.543449 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d0bab13-bd2b-4494-9177-7709e5a27d3c-hub\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.543570 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.543475 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkgp8\" (UniqueName: \"kubernetes.io/projected/5760faea-8ad3-45fc-b57f-25337f6b027e-kube-api-access-lkgp8\") pod \"klusterlet-addon-workmgr-797bb4bf7f-tbtsc\" (UID: \"5760faea-8ad3-45fc-b57f-25337f6b027e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" Jul 13 14:21:37.543570 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.543504 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d0bab13-bd2b-4494-9177-7709e5a27d3c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.543570 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.543531 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d0bab13-bd2b-4494-9177-7709e5a27d3c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.543570 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:37.543549 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:21:37.543894 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.543585 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75jvp\" (UniqueName: \"kubernetes.io/projected/2d0bab13-bd2b-4494-9177-7709e5a27d3c-kube-api-access-75jvp\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.543894 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:37.543620 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls podName:bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe nodeName:}" failed. No retries permitted until 2026-07-13 14:21:39.543599892 +0000 UTC m=+41.228020206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z78ln" (UID: "bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe") : secret "insights-runtime-extractor-tls" not found Jul 13 14:21:37.543894 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.543659 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:37.543894 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.543688 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d0bab13-bd2b-4494-9177-7709e5a27d3c-ca\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.543894 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.543726 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5760faea-8ad3-45fc-b57f-25337f6b027e-tmp\") pod \"klusterlet-addon-workmgr-797bb4bf7f-tbtsc\" (UID: \"5760faea-8ad3-45fc-b57f-25337f6b027e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" Jul 13 14:21:37.543894 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.543750 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5760faea-8ad3-45fc-b57f-25337f6b027e-klusterlet-config\") pod \"klusterlet-addon-workmgr-797bb4bf7f-tbtsc\" (UID: \"5760faea-8ad3-45fc-b57f-25337f6b027e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" Jul 13 14:21:37.544176 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:37.543979 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 14:21:37.544176 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:37.544051 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-rbac-proxy-cm podName:bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe nodeName:}" failed. No retries permitted until 2026-07-13 14:21:39.544031545 +0000 UTC m=+41.228451844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z78ln" (UID: "bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe") : configmap "kube-rbac-proxy" not found Jul 13 14:21:37.544286 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.544188 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0bab13-bd2b-4494-9177-7709e5a27d3c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.544413 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.544356 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5760faea-8ad3-45fc-b57f-25337f6b027e-tmp\") pod \"klusterlet-addon-workmgr-797bb4bf7f-tbtsc\" (UID: \"5760faea-8ad3-45fc-b57f-25337f6b027e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" Jul 13 14:21:37.546759 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.546737 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d0bab13-bd2b-4494-9177-7709e5a27d3c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.546881 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.546771 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d0bab13-bd2b-4494-9177-7709e5a27d3c-ca\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.546881 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.546851 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d0bab13-bd2b-4494-9177-7709e5a27d3c-hub\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.546881 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.546850 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5760faea-8ad3-45fc-b57f-25337f6b027e-klusterlet-config\") pod \"klusterlet-addon-workmgr-797bb4bf7f-tbtsc\" (UID: \"5760faea-8ad3-45fc-b57f-25337f6b027e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" Jul 13 14:21:37.547035 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.547017 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d0bab13-bd2b-4494-9177-7709e5a27d3c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.561903 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.561876 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75jvp\" (UniqueName: \"kubernetes.io/projected/2d0bab13-bd2b-4494-9177-7709e5a27d3c-kube-api-access-75jvp\") pod \"cluster-proxy-proxy-agent-77ccbfc548-p45hh\" (UID: \"2d0bab13-bd2b-4494-9177-7709e5a27d3c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.562735 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.562711 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkgp8\" (UniqueName: \"kubernetes.io/projected/5760faea-8ad3-45fc-b57f-25337f6b027e-kube-api-access-lkgp8\") pod \"klusterlet-addon-workmgr-797bb4bf7f-tbtsc\" (UID: \"5760faea-8ad3-45fc-b57f-25337f6b027e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" Jul 13 14:21:37.636614 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.636538 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xcv69_91eed05b-8ccb-4806-ac18-2ced768c9780/dns-node-resolver/0.log" Jul 13 14:21:37.648680 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.648651 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:37.703936 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.703909 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" Jul 13 14:21:37.732073 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.732037 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:21:37.756120 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:37.756084 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:38.226170 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:38.226145 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-76bmg_5962361d-1783-4050-8470-a1b9518399bb/node-ca/0.log" Jul 13 14:21:38.475419 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:38.475255 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh"] Jul 13 14:21:38.479046 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:38.479016 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d0bab13_bd2b_4494_9177_7709e5a27d3c.slice/crio-324a8da55dc439694f6f79153c0aa35f1f2906b14c2a57e817c603ec882cbdd9 WatchSource:0}: Error finding container 324a8da55dc439694f6f79153c0aa35f1f2906b14c2a57e817c603ec882cbdd9: Status 404 returned error can't find the container with id 324a8da55dc439694f6f79153c0aa35f1f2906b14c2a57e817c603ec882cbdd9 Jul 13 14:21:38.488429 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:38.488390 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:38.498114 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:38.497986 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bj582"] Jul 13 14:21:38.499840 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:38.499814 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc"] Jul 13 14:21:38.501875 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:38.501833 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a63d66_f350_425c_b173_98c532394025.slice/crio-896c9831326c15524588061f09e688a3bca34ae8b17ecc684344b6689af5d018 WatchSource:0}: Error finding container 896c9831326c15524588061f09e688a3bca34ae8b17ecc684344b6689af5d018: Status 404 returned error can't find the container with id 896c9831326c15524588061f09e688a3bca34ae8b17ecc684344b6689af5d018 Jul 13 14:21:38.502979 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:38.502946 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5760faea_8ad3_45fc_b57f_25337f6b027e.slice/crio-c78a979a40315c06836bc8a6a2dc03974ad414365f307681322c3145adfb37e2 WatchSource:0}: Error finding container c78a979a40315c06836bc8a6a2dc03974ad414365f307681322c3145adfb37e2: Status 404 returned error can't find the container with id c78a979a40315c06836bc8a6a2dc03974ad414365f307681322c3145adfb37e2 Jul 13 14:21:38.614534 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:38.614507 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:38.692271 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:38.692239 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:39.020753 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:39.020722 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6dmbf" event={"ID":"ea289c9c-fdbc-4830-ad88-6d6fbf43834d","Type":"ContainerStarted","Data":"25c2b264eefe9cb507893733c7e32c84ee25988ee3f2b11a12d023d05877eacc"} Jul 13 14:21:39.023314 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:39.023289 2566 generic.go:356] "Generic (PLEG): container finished" podID="a0d488fc-58f7-409e-ba15-7f7980cbd394" containerID="8403624d08b13a554892cee5023f715426cfe4e64932ad1b99f34b34de28cb25" exitCode=0 Jul 13 14:21:39.023427 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:39.023372 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbm2b" event={"ID":"a0d488fc-58f7-409e-ba15-7f7980cbd394","Type":"ContainerDied","Data":"8403624d08b13a554892cee5023f715426cfe4e64932ad1b99f34b34de28cb25"} Jul 13 14:21:39.024468 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:39.024446 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" event={"ID":"2d0bab13-bd2b-4494-9177-7709e5a27d3c","Type":"ContainerStarted","Data":"324a8da55dc439694f6f79153c0aa35f1f2906b14c2a57e817c603ec882cbdd9"} Jul 13 14:21:39.025478 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:39.025453 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bj582" event={"ID":"88a63d66-f350-425c-b173-98c532394025","Type":"ContainerStarted","Data":"896c9831326c15524588061f09e688a3bca34ae8b17ecc684344b6689af5d018"} Jul 13 14:21:39.026502 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:39.026484 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" event={"ID":"5760faea-8ad3-45fc-b57f-25337f6b027e","Type":"ContainerStarted","Data":"c78a979a40315c06836bc8a6a2dc03974ad414365f307681322c3145adfb37e2"} Jul 13 14:21:39.034591 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:39.034576 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6dmbf_ea289c9c-fdbc-4830-ad88-6d6fbf43834d/serve-healthcheck-canary/0.log" Jul 13 14:21:39.050052 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:39.050016 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6dmbf" podStartSLOduration=1.49875194 podStartE2EDuration="6.050003757s" totalImagesPullingTime="4.551251817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:39.049420588 +0000 UTC m=+40.733840922" watchObservedRunningTime="2026-07-13 14:21:39.050003757 +0000 UTC m=+40.734424062" Jul 13 14:21:39.563130 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:39.563092 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:39.563601 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:39.563281 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:21:39.563601 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:39.563306 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:39.563601 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:39.563357 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls podName:bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe nodeName:}" failed. No retries permitted until 2026-07-13 14:21:43.563336674 +0000 UTC m=+45.247756960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z78ln" (UID: "bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe") : secret "insights-runtime-extractor-tls" not found Jul 13 14:21:39.563601 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:39.563401 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 14:21:39.563601 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:39.563465 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-rbac-proxy-cm podName:bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe nodeName:}" failed. No retries permitted until 2026-07-13 14:21:43.563447355 +0000 UTC m=+45.247867657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-z78ln" (UID: "bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe") : configmap "kube-rbac-proxy" not found Jul 13 14:21:39.724373 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:39.724337 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:39.961617 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:39.961498 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:40.037085 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:40.035712 2566 generic.go:356] "Generic (PLEG): container finished" podID="a0d488fc-58f7-409e-ba15-7f7980cbd394" containerID="e3e919af052d70070c584685b6af815029d7ee0c9c5cd94647fb28eab4fb9089" exitCode=0 Jul 13 14:21:40.037085 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:40.036814 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbm2b" event={"ID":"a0d488fc-58f7-409e-ba15-7f7980cbd394","Type":"ContainerDied","Data":"e3e919af052d70070c584685b6af815029d7ee0c9c5cd94647fb28eab4fb9089"} Jul 13 14:21:40.940227 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:40.940149 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:41.045841 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:41.045086 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qbm2b" event={"ID":"a0d488fc-58f7-409e-ba15-7f7980cbd394","Type":"ContainerStarted","Data":"8bf17b212cadd974ea6e78580db4a2e4c18fadbbbb8e458e055597313c74d8bb"} Jul 13 14:21:41.077988 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:41.077398 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:41.077988 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:41.077506 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:41.077988 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:41.077582 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 14:21:41.077988 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:41.077645 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls podName:58085733-1f46-4db6-9ae7-1eae8fec5bab nodeName:}" failed. No retries permitted until 2026-07-13 14:21:49.077625578 +0000 UTC m=+50.762045862 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls") pod "dns-default-bhvr5" (UID: "58085733-1f46-4db6-9ae7-1eae8fec5bab") : secret "dns-default-metrics-tls" not found Jul 13 14:21:41.077988 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:41.077809 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 14:21:41.077988 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:41.077825 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5647674757-gwgns: secret "image-registry-tls" not found Jul 13 14:21:41.077988 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:41.077874 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls podName:b9f13eee-31c5-40f5-a420-c1214e371eb2 nodeName:}" failed. No retries permitted until 2026-07-13 14:21:49.077859056 +0000 UTC m=+50.762279354 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls") pod "image-registry-5647674757-gwgns" (UID: "b9f13eee-31c5-40f5-a420-c1214e371eb2") : secret "image-registry-tls" not found Jul 13 14:21:41.228842 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:41.228727 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:41.327377 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:41.327332 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:41.489888 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:41.489800 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:41.598100 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:41.598070 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:41.687089 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:41.687055 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:42.049156 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:42.049122 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:43.601442 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:43.601400 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:43.601866 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:43.601578 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:43.601866 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:43.601681 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 14:21:43.601866 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:21:43.601744 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls podName:bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe nodeName:}" failed. No retries permitted until 2026-07-13 14:21:51.601723775 +0000 UTC m=+53.286144071 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-z78ln" (UID: "bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe") : secret "insights-runtime-extractor-tls" not found Jul 13 14:21:43.602084 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:43.602063 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:44.112737 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:44.112698 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:44.220503 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:44.220460 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:44.623384 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:44.623356 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:44.799723 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:44.799698 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:44.927867 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:44.927721 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:45.057258 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.057221 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" event={"ID":"2d0bab13-bd2b-4494-9177-7709e5a27d3c","Type":"ContainerStarted","Data":"b2445dd30afaa5ef6243d4d150d052cc6c29407a33e3f36e5fb6691c7b86b907"} Jul 13 14:21:45.057392 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.057268 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" event={"ID":"2d0bab13-bd2b-4494-9177-7709e5a27d3c","Type":"ContainerStarted","Data":"cab2c878717f20d3c5490016ffa93c3b5b8e8f22dd14c4a584d1182284d99f2c"} Jul 13 14:21:45.057392 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.057286 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" event={"ID":"2d0bab13-bd2b-4494-9177-7709e5a27d3c","Type":"ContainerStarted","Data":"b48bc8871aadb42d538c7a1b09da2b555bcdc9545f0e73b2c188cef35aca4e1b"} Jul 13 14:21:45.059006 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.058975 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bj582" event={"ID":"88a63d66-f350-425c-b173-98c532394025","Type":"ContainerStarted","Data":"11d57c3699efc34acb8d60fe08afa3dfaefbd76b086900656d151d77becd4e5e"} Jul 13 14:21:45.059099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.059034 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bj582" event={"ID":"88a63d66-f350-425c-b173-98c532394025","Type":"ContainerStarted","Data":"f81620c52409171878b5f1334cb5dfdf0952ceeafe5c608d5cdbaf81ac1454f3"} Jul 13 14:21:45.060411 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.060389 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" event={"ID":"5760faea-8ad3-45fc-b57f-25337f6b027e","Type":"ContainerStarted","Data":"a73198551efdbe374915cd9a94791294955a94321797095b88e5d7e27fc2f150"} Jul 13 14:21:45.060575 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.060553 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" Jul 13 14:21:45.062092 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.062076 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" Jul 13 14:21:45.089121 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.089048 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qbm2b" podStartSLOduration=11.668916061000001 podStartE2EDuration="46.089036438s" totalImagesPullingTime="34.420120377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:20:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:41.079760649 +0000 UTC m=+42.764180960" watchObservedRunningTime="2026-07-13 14:21:45.089036438 +0000 UTC m=+46.773456745" Jul 13 14:21:45.089806 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.089749 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" podStartSLOduration=1.9638220899999999 podStartE2EDuration="8.08974026s" totalImagesPullingTime="6.12591817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:45.08717935 +0000 UTC m=+46.771599673" watchObservedRunningTime="2026-07-13 14:21:45.08974026 +0000 UTC m=+46.774160602" Jul 13 14:21:45.109835 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.109777 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bj582" podStartSLOduration=3.048367749 podStartE2EDuration="9.109767552s" totalImagesPullingTime="6.061399803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:45.109249982 +0000 UTC m=+46.793670310" watchObservedRunningTime="2026-07-13 14:21:45.109767552 +0000 UTC m=+46.794187836" Jul 13 14:21:45.129458 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.129424 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-797bb4bf7f-tbtsc" podStartSLOduration=2.042229226 podStartE2EDuration="8.129412757s" totalImagesPullingTime="6.087183531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:45.128625473 +0000 UTC m=+46.813045788" watchObservedRunningTime="2026-07-13 14:21:45.129412757 +0000 UTC m=+46.813833063" Jul 13 14:21:45.432452 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.432376 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:45.493768 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.493742 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:45.630510 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.630461 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:45.821941 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:45.821918 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:46.008724 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:46.008675 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:46.236671 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:46.236570 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:46.504966 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:46.504860 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:47.734410 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:47.734365 2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" podUID="2d0bab13-bd2b-4494-9177-7709e5a27d3c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 13 14:21:47.915914 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:47.915858 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:48.016359 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:48.016279 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:48.115905 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:48.115866 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:48.217570 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:48.217536 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:48.389003 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:48.388954 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:49.022067 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:49.022028 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:49.148243 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:49.148215 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:49.148411 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:49.148275 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:49.150587 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:49.150552 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58085733-1f46-4db6-9ae7-1eae8fec5bab-metrics-tls\") pod \"dns-default-bhvr5\" (UID: \"58085733-1f46-4db6-9ae7-1eae8fec5bab\") " pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:49.150696 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:49.150598 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls\") pod \"image-registry-5647674757-gwgns\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:49.160508 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:49.160491 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:49.181774 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:49.181753 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:49.323338 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:49.323310 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5647674757-gwgns"] Jul 13 14:21:49.326224 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:49.326200 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f13eee_31c5_40f5_a420_c1214e371eb2.slice/crio-b972e54af37143318a70ab40b79a3a3d468437eb740786e026f6337236ebb700 WatchSource:0}: Error finding container b972e54af37143318a70ab40b79a3a3d468437eb740786e026f6337236ebb700: Status 404 returned error can't find the container with id b972e54af37143318a70ab40b79a3a3d468437eb740786e026f6337236ebb700 Jul 13 14:21:49.332335 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:49.332307 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bhvr5"] Jul 13 14:21:49.334857 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:49.334838 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58085733_1f46_4db6_9ae7_1eae8fec5bab.slice/crio-fd00a23f4370498e94de2df0e9af1fabf06c4b45cee145b84104e54669b8bab9 WatchSource:0}: Error finding container fd00a23f4370498e94de2df0e9af1fabf06c4b45cee145b84104e54669b8bab9: Status 404 returned error can't find the container with id fd00a23f4370498e94de2df0e9af1fabf06c4b45cee145b84104e54669b8bab9 Jul 13 14:21:49.439111 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:49.439071 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:49.632154 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:49.632119 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:49.818348 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:49.818310 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:49.895170 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:49.895093 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:50.025004 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:50.024974 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:50.080126 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:50.080089 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bhvr5" event={"ID":"58085733-1f46-4db6-9ae7-1eae8fec5bab","Type":"ContainerStarted","Data":"fd00a23f4370498e94de2df0e9af1fabf06c4b45cee145b84104e54669b8bab9"} Jul 13 14:21:50.081547 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:50.081521 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5647674757-gwgns" event={"ID":"b9f13eee-31c5-40f5-a420-c1214e371eb2","Type":"ContainerStarted","Data":"eed16fcd5aea99755627686e31617c010bd4402f2e435c7d2b3c38a8d2f2abd1"} Jul 13 14:21:50.081670 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:50.081558 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5647674757-gwgns" event={"ID":"b9f13eee-31c5-40f5-a420-c1214e371eb2","Type":"ContainerStarted","Data":"b972e54af37143318a70ab40b79a3a3d468437eb740786e026f6337236ebb700"} Jul 13 14:21:50.081670 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:50.081653 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:21:50.770506 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:50.770473 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:50.872063 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:50.872032 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:51.021908 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:51.021826 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:51.265723 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:51.265685 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:51.430590 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:51.430557 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:51.667515 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:51.667485 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:51.669728 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:51.669703 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-z78ln\" (UID: \"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe\") " pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:51.800906 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:51.800872 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-z78ln" Jul 13 14:21:51.914814 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:51.914773 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:51.952824 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:51.952741 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5647674757-gwgns" podStartSLOduration=26.952718629 podStartE2EDuration="26.952718629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:50.105004372 +0000 UTC m=+51.789424677" watchObservedRunningTime="2026-07-13 14:21:51.952718629 +0000 UTC m=+53.637138936" Jul 13 14:21:51.953217 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:51.953098 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-z78ln"] Jul 13 14:21:51.956390 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:21:51.956349 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfeeed1f_6ca0_481d_bd28_e2c4aaf97abe.slice/crio-18d9e99ab6e15089bbf382389e21b198c2fe97ebbe1515b7edadfe80f69008fb WatchSource:0}: Error finding container 18d9e99ab6e15089bbf382389e21b198c2fe97ebbe1515b7edadfe80f69008fb: Status 404 returned error can't find the container with id 18d9e99ab6e15089bbf382389e21b198c2fe97ebbe1515b7edadfe80f69008fb Jul 13 14:21:52.024809 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:52.024722 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:52.089452 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:52.089419 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bhvr5" event={"ID":"58085733-1f46-4db6-9ae7-1eae8fec5bab","Type":"ContainerStarted","Data":"8e191f8c1b638e701e797ebe54919256f862be165728199f9201df410dfe5ab3"} Jul 13 14:21:52.089615 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:52.089477 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bhvr5" event={"ID":"58085733-1f46-4db6-9ae7-1eae8fec5bab","Type":"ContainerStarted","Data":"9d031e8dab6ac5831aaf0b7f67ccbb7945aa1df89c81abd25d0b0ae7cf7682c0"} Jul 13 14:21:52.089615 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:52.089526 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bhvr5" Jul 13 14:21:52.090436 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:52.090414 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z78ln" event={"ID":"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe","Type":"ContainerStarted","Data":"18d9e99ab6e15089bbf382389e21b198c2fe97ebbe1515b7edadfe80f69008fb"} Jul 13 14:21:52.111207 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:52.111164 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bhvr5" podStartSLOduration=16.672179434 podStartE2EDuration="19.111150829s" totalImagesPullingTime="2.438971395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:21:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:52.109691526 +0000 UTC m=+53.794111841" watchObservedRunningTime="2026-07-13 14:21:52.111150829 +0000 UTC m=+53.795571135" Jul 13 14:21:52.412622 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:52.412581 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:52.493040 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:52.493002 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:21:56.110061 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:56.110027 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z78ln" event={"ID":"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe","Type":"ContainerStarted","Data":"87bf14ca5ac3acd700291b75bedd65170557f4136954f8956d59550d40bc32b3"} Jul 13 14:21:57.115717 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:57.115670 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z78ln" event={"ID":"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe","Type":"ContainerStarted","Data":"fd1ee8c793dd53491102d7adbc76ffd7ff4bd78cbc756bf82cef90f581a60423"} Jul 13 14:21:57.733584 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:57.733542 2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" podUID="2d0bab13-bd2b-4494-9177-7709e5a27d3c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 13 14:21:59.124080 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:59.124044 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-z78ln" event={"ID":"bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe","Type":"ContainerStarted","Data":"bc4ca6bb05eeaa56b90f252aa538b3ac8dc6d4c238eedffae59fa9ea6ee53fe7"} Jul 13 14:21:59.145741 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:21:59.145695 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-z78ln" podStartSLOduration=17.17362879 podStartE2EDuration="24.145682725s" totalImagesPullingTime="6.972053935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:21:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:21:59.145487978 +0000 UTC m=+60.829908304" watchObservedRunningTime="2026-07-13 14:21:59.145682725 +0000 UTC m=+60.830103028" Jul 13 14:22:00.009138 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:00.009111 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pr6tf" Jul 13 14:22:00.960207 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:00.960178 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5647674757-gwgns"] Jul 13 14:22:00.978743 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:00.978711 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f9447d85f-z292b"] Jul 13 14:22:00.981553 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:00.981532 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:00.986920 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:00.986892 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 13 14:22:00.987722 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:00.987683 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 13 14:22:00.987831 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:00.987773 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d5w6n\"" Jul 13 14:22:00.987904 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:00.987833 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 13 14:22:00.989317 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:00.989302 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 13 14:22:00.989382 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:00.989369 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 13 14:22:00.989588 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:00.989575 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 13 14:22:00.999621 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:00.999603 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 13 14:22:01.016328 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.016299 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f9447d85f-z292b"] Jul 13 14:22:01.046544 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.046513 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ccx\" (UniqueName: \"kubernetes.io/projected/4889381b-b506-4b1a-8af5-9a10edb4ce98-kube-api-access-p8ccx\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.046661 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.046552 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-config\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.046661 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.046633 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-serving-cert\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.046740 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.046667 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-oauth-config\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.046740 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.046697 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-oauth-serving-cert\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.046840 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.046828 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-service-ca\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.100880 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.100845 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd"] Jul 13 14:22:01.102773 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.102753 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.112075 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.112025 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:01.126947 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.126921 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd"] Jul 13 14:22:01.147264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.147242 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-oauth-serving-cert\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.147372 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.147292 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-service-ca\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.147372 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.147312 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ccx\" (UniqueName: \"kubernetes.io/projected/4889381b-b506-4b1a-8af5-9a10edb4ce98-kube-api-access-p8ccx\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.147480 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.147428 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-config\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.147544 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.147525 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-serving-cert\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.147597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.147561 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-oauth-config\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.147889 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.147867 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-service-ca\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.148090 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.148071 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-config\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.148304 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.148286 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-oauth-serving-cert\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.150243 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.150212 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-serving-cert\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.150586 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.150568 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-oauth-config\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.165543 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.165524 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ccx\" (UniqueName: \"kubernetes.io/projected/4889381b-b506-4b1a-8af5-9a10edb4ce98-kube-api-access-p8ccx\") pod \"console-7f9447d85f-z292b\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.248134 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.248068 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7aab2ee-40cf-473b-b318-f1b61ca71070-registry-tls\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.248134 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.248100 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7aab2ee-40cf-473b-b318-f1b61ca71070-image-registry-private-configuration\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.248134 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.248123 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7aab2ee-40cf-473b-b318-f1b61ca71070-registry-certificates\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.248300 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.248240 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7aab2ee-40cf-473b-b318-f1b61ca71070-installation-pull-secrets\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.248300 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.248275 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7aab2ee-40cf-473b-b318-f1b61ca71070-bound-sa-token\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.248362 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.248300 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66f22\" (UniqueName: \"kubernetes.io/projected/f7aab2ee-40cf-473b-b318-f1b61ca71070-kube-api-access-66f22\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.248362 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.248341 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7aab2ee-40cf-473b-b318-f1b61ca71070-ca-trust-extracted\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.248424 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.248374 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7aab2ee-40cf-473b-b318-f1b61ca71070-trusted-ca\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.289901 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.289875 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:01.348845 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.348811 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7aab2ee-40cf-473b-b318-f1b61ca71070-ca-trust-extracted\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.348969 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.348869 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7aab2ee-40cf-473b-b318-f1b61ca71070-trusted-ca\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.348969 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.348921 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7aab2ee-40cf-473b-b318-f1b61ca71070-registry-tls\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.348969 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.348945 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7aab2ee-40cf-473b-b318-f1b61ca71070-image-registry-private-configuration\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.349075 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.348969 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7aab2ee-40cf-473b-b318-f1b61ca71070-registry-certificates\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.349075 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.349037 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7aab2ee-40cf-473b-b318-f1b61ca71070-installation-pull-secrets\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.349075 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.349070 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7aab2ee-40cf-473b-b318-f1b61ca71070-bound-sa-token\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.349210 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.349099 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66f22\" (UniqueName: \"kubernetes.io/projected/f7aab2ee-40cf-473b-b318-f1b61ca71070-kube-api-access-66f22\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.349563 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.349534 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f7aab2ee-40cf-473b-b318-f1b61ca71070-ca-trust-extracted\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.350549 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.350498 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7aab2ee-40cf-473b-b318-f1b61ca71070-trusted-ca\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.350734 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.350711 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f7aab2ee-40cf-473b-b318-f1b61ca71070-registry-certificates\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.352531 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.352427 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f7aab2ee-40cf-473b-b318-f1b61ca71070-installation-pull-secrets\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.353099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.353072 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f7aab2ee-40cf-473b-b318-f1b61ca71070-registry-tls\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.353189 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.353159 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f7aab2ee-40cf-473b-b318-f1b61ca71070-image-registry-private-configuration\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.359004 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.358974 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f7aab2ee-40cf-473b-b318-f1b61ca71070-bound-sa-token\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.359549 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.359530 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66f22\" (UniqueName: \"kubernetes.io/projected/f7aab2ee-40cf-473b-b318-f1b61ca71070-kube-api-access-66f22\") pod \"image-registry-5d7bc5cf5c-gwtrd\" (UID: \"f7aab2ee-40cf-473b-b318-f1b61ca71070\") " pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.411987 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.411958 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:01.418033 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.418004 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f9447d85f-z292b"] Jul 13 14:22:01.421015 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:22:01.420989 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4889381b_b506_4b1a_8af5_9a10edb4ce98.slice/crio-ba9c6fcc091eb36ca887eadfbddbaa12685e861d861e32417fcf553558f5873c WatchSource:0}: Error finding container ba9c6fcc091eb36ca887eadfbddbaa12685e861d861e32417fcf553558f5873c: Status 404 returned error can't find the container with id ba9c6fcc091eb36ca887eadfbddbaa12685e861d861e32417fcf553558f5873c Jul 13 14:22:01.541096 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.541024 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd"] Jul 13 14:22:01.544191 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:22:01.544168 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7aab2ee_40cf_473b_b318_f1b61ca71070.slice/crio-05165a618bc0259d53ba49bc6c9d3fd7bacb6f2280056c360181505a5cdd89a6 WatchSource:0}: Error finding container 05165a618bc0259d53ba49bc6c9d3fd7bacb6f2280056c360181505a5cdd89a6: Status 404 returned error can't find the container with id 05165a618bc0259d53ba49bc6c9d3fd7bacb6f2280056c360181505a5cdd89a6 Jul 13 14:22:01.549830 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.549807 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:01.644912 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:01.644887 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:02.096351 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:02.096323 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bhvr5" Jul 13 14:22:02.101496 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:02.101470 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:02.134258 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:02.134224 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" event={"ID":"f7aab2ee-40cf-473b-b318-f1b61ca71070","Type":"ContainerStarted","Data":"11d9505f956257cccc76ee8e4ad4f0f8697c328bd0aa6a539442dd6684cf22a1"} Jul 13 14:22:02.134413 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:02.134284 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" event={"ID":"f7aab2ee-40cf-473b-b318-f1b61ca71070","Type":"ContainerStarted","Data":"05165a618bc0259d53ba49bc6c9d3fd7bacb6f2280056c360181505a5cdd89a6"} Jul 13 14:22:02.134413 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:02.134311 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:02.135736 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:02.135712 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9447d85f-z292b" event={"ID":"4889381b-b506-4b1a-8af5-9a10edb4ce98","Type":"ContainerStarted","Data":"ba9c6fcc091eb36ca887eadfbddbaa12685e861d861e32417fcf553558f5873c"} Jul 13 14:22:02.157801 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:02.157748 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" podStartSLOduration=1.157733414 podStartE2EDuration="1.157733414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:22:02.157077757 +0000 UTC m=+63.841498073" watchObservedRunningTime="2026-07-13 14:22:02.157733414 +0000 UTC m=+63.842153720" Jul 13 14:22:04.576954 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.576908 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:22:04.577379 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.576993 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs\") pod \"network-metrics-daemon-cj24h\" (UID: \"cde58de7-b1b7-41c5-83c1-eaed967537b2\") " pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:22:04.579797 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.579762 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 14:22:04.579899 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.579762 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 14:22:04.590378 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.590351 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b-original-pull-secret\") pod \"global-pull-secret-syncer-f9wtx\" (UID: \"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b\") " pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:22:04.590489 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.590358 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde58de7-b1b7-41c5-83c1-eaed967537b2-metrics-certs\") pod \"network-metrics-daemon-cj24h\" (UID: \"cde58de7-b1b7-41c5-83c1-eaed967537b2\") " pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:22:04.677366 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.677336 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgkkb\" (UniqueName: \"kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb\") pod \"network-check-target-sl54c\" (UID: \"4629c8d6-afb7-43da-b676-9d6418414491\") " pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:22:04.680588 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.680571 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 14:22:04.691218 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.691198 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 14:22:04.700952 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.700929 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgkkb\" (UniqueName: \"kubernetes.io/projected/4629c8d6-afb7-43da-b676-9d6418414491-kube-api-access-jgkkb\") pod \"network-check-target-sl54c\" (UID: \"4629c8d6-afb7-43da-b676-9d6418414491\") " pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:22:04.774621 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.774589 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f9wtx" Jul 13 14:22:04.784490 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.784465 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hslvn\"" Jul 13 14:22:04.789274 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.789251 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5z6w6\"" Jul 13 14:22:04.792534 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.792515 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:22:04.797414 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.797394 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cj24h" Jul 13 14:22:04.829995 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.829885 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:04.929250 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.929141 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:04.938963 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.938936 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f9wtx"] Jul 13 14:22:04.945436 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:22:04.945406 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1b73e5_52d4_43b9_b2eb_151cb0a5b16b.slice/crio-2c043f979cf51bf0e320063ce2014447132335df1c7d059b651af37e83da34ae WatchSource:0}: Error finding container 2c043f979cf51bf0e320063ce2014447132335df1c7d059b651af37e83da34ae: Status 404 returned error can't find the container with id 2c043f979cf51bf0e320063ce2014447132335df1c7d059b651af37e83da34ae Jul 13 14:22:04.963694 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.963665 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sl54c"] Jul 13 14:22:04.967954 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:22:04.967915 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4629c8d6_afb7_43da_b676_9d6418414491.slice/crio-f6c78cc3bdeb1b0914c3241f04d2399016fa50722fee554edae015a2527d5f5d WatchSource:0}: Error finding container f6c78cc3bdeb1b0914c3241f04d2399016fa50722fee554edae015a2527d5f5d: Status 404 returned error can't find the container with id f6c78cc3bdeb1b0914c3241f04d2399016fa50722fee554edae015a2527d5f5d Jul 13 14:22:04.993191 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:04.993161 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cj24h"] Jul 13 14:22:04.997038 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:22:04.997010 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde58de7_b1b7_41c5_83c1_eaed967537b2.slice/crio-639e903ad569aa3c2dff54b81579901e2642e18ebfd9fdbd823dead1bc34713f WatchSource:0}: Error finding container 639e903ad569aa3c2dff54b81579901e2642e18ebfd9fdbd823dead1bc34713f: Status 404 returned error can't find the container with id 639e903ad569aa3c2dff54b81579901e2642e18ebfd9fdbd823dead1bc34713f Jul 13 14:22:05.010145 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:05.010084 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:05.148062 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:05.147980 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cj24h" event={"ID":"cde58de7-b1b7-41c5-83c1-eaed967537b2","Type":"ContainerStarted","Data":"639e903ad569aa3c2dff54b81579901e2642e18ebfd9fdbd823dead1bc34713f"} Jul 13 14:22:05.149192 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:05.149166 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sl54c" event={"ID":"4629c8d6-afb7-43da-b676-9d6418414491","Type":"ContainerStarted","Data":"f6c78cc3bdeb1b0914c3241f04d2399016fa50722fee554edae015a2527d5f5d"} Jul 13 14:22:05.150221 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:05.150200 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f9wtx" event={"ID":"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b","Type":"ContainerStarted","Data":"2c043f979cf51bf0e320063ce2014447132335df1c7d059b651af37e83da34ae"} Jul 13 14:22:05.542059 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:05.541963 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:05.771351 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:05.771315 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:05.839273 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:05.839232 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:06.426330 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:06.426308 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:06.488976 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:06.488945 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:07.159393 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:07.159359 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9447d85f-z292b" event={"ID":"4889381b-b506-4b1a-8af5-9a10edb4ce98","Type":"ContainerStarted","Data":"02b211f8932324c02b2ed1cd828d42f04182357d3ecb4aab46eb6dbc6a8e68ba"} Jul 13 14:22:07.180815 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:07.180184 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f9447d85f-z292b" podStartSLOduration=2.198166843 podStartE2EDuration="7.180163126s" totalImagesPullingTime="4.981996283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:22:07.179722314 +0000 UTC m=+68.864142645" watchObservedRunningTime="2026-07-13 14:22:07.180163126 +0000 UTC m=+68.864583433" Jul 13 14:22:07.733855 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:07.733815 2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" podUID="2d0bab13-bd2b-4494-9177-7709e5a27d3c" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 13 14:22:07.734033 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:07.733899 2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" Jul 13 14:22:07.734485 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:07.734443 2566 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"b2445dd30afaa5ef6243d4d150d052cc6c29407a33e3f36e5fb6691c7b86b907"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 13 14:22:07.734575 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:07.734512 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" podUID="2d0bab13-bd2b-4494-9177-7709e5a27d3c" containerName="service-proxy" containerID="cri-o://b2445dd30afaa5ef6243d4d150d052cc6c29407a33e3f36e5fb6691c7b86b907" gracePeriod=30 Jul 13 14:22:08.077578 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:08.077547 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:08.165495 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:08.165445 2566 generic.go:356] "Generic (PLEG): container finished" podID="2d0bab13-bd2b-4494-9177-7709e5a27d3c" containerID="b2445dd30afaa5ef6243d4d150d052cc6c29407a33e3f36e5fb6691c7b86b907" exitCode=2 Jul 13 14:22:08.165949 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:08.165518 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" event={"ID":"2d0bab13-bd2b-4494-9177-7709e5a27d3c","Type":"ContainerDied","Data":"b2445dd30afaa5ef6243d4d150d052cc6c29407a33e3f36e5fb6691c7b86b907"} Jul 13 14:22:08.165949 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:08.165570 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-77ccbfc548-p45hh" event={"ID":"2d0bab13-bd2b-4494-9177-7709e5a27d3c","Type":"ContainerStarted","Data":"8a76d161a93baabe35cc9b5d5c0ade3b5a150c6063486e9103091edb5135db84"} Jul 13 14:22:08.167697 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:08.167671 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cj24h" event={"ID":"cde58de7-b1b7-41c5-83c1-eaed967537b2","Type":"ContainerStarted","Data":"36529d0c04dd15cf8836a79f32abe233054d71ad676bf681c28804a6109a8c4a"} Jul 13 14:22:08.167833 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:08.167727 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cj24h" event={"ID":"cde58de7-b1b7-41c5-83c1-eaed967537b2","Type":"ContainerStarted","Data":"c210ff86cd11af63d5a9f2dc0087e8eb0f9103f3a1ad1e9478869240995b8e61"} Jul 13 14:22:08.208346 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:08.208235 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cj24h" podStartSLOduration=67.116830593 podStartE2EDuration="1m9.208210043s" totalImagesPullingTime="2.09137945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:22:08.206927667 +0000 UTC m=+69.891347998" watchObservedRunningTime="2026-07-13 14:22:08.208210043 +0000 UTC m=+69.892630350" Jul 13 14:22:08.241837 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:08.241803 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:08.335263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:08.335158 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:08.435612 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:08.435568 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:10.044833 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:10.044778 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:10.138664 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:10.138636 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:10.611847 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:10.611821 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:10.967270 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:10.967244 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:22:11.179996 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:11.179963 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sl54c" event={"ID":"4629c8d6-afb7-43da-b676-9d6418414491","Type":"ContainerStarted","Data":"ac9a02051bace3e52d162788f6818e95a7bd41856406ccfdb67640f130b90688"} Jul 13 14:22:11.180437 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:11.180083 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:22:11.181557 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:11.181530 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f9wtx" event={"ID":"1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b","Type":"ContainerStarted","Data":"56e9a18ef2b26bc3bd1473ad3bb50ca92a029362779a74ccb9f064bc1633aa78"} Jul 13 14:22:11.199232 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:11.199188 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sl54c" podStartSLOduration=67.980058514 podStartE2EDuration="1m12.199171136s" totalImagesPullingTime="4.219112622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:22:11.197901296 +0000 UTC m=+72.882321602" watchObservedRunningTime="2026-07-13 14:22:11.199171136 +0000 UTC m=+72.883591441" Jul 13 14:22:11.215318 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:11.215278 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f9wtx" podStartSLOduration=66.54962825 podStartE2EDuration="1m12.215264964s" totalImagesPullingTime="5.665636714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 14:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:22:11.214760393 +0000 UTC m=+72.899180700" watchObservedRunningTime="2026-07-13 14:22:11.215264964 +0000 UTC m=+72.899685270" Jul 13 14:22:11.265562 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:11.265493 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:11.290130 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:11.290092 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:11.290221 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:11.290150 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:11.295015 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:11.294996 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:11.365956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:11.365927 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:11.709151 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:11.709123 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f9447d85f-z292b"] Jul 13 14:22:11.875734 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:11.875705 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:11.975978 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:11.975908 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:12.188557 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:12.188524 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:12.360835 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:12.360804 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:12.462807 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:12.462712 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:12.564141 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:12.564109 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:12.663346 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:12.663273 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:12.790637 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:12.790610 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:13.090755 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:13.090724 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:14.889936 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:14.889904 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:15.034918 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:15.034884 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:15.136518 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:15.136485 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:15.237025 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:15.236930 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:15.569600 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:15.569568 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:17.368960 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:17.368915 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:17.470086 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:17.470049 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:17.713067 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:17.712964 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:19.124698 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:19.124669 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:19.478587 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:19.478459 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:19.554991 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:19.554958 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:20.264054 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:20.263882 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:20.363911 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:20.363879 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:20.778597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:20.778557 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:20.848283 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:20.848256 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:23.143152 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:23.143123 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d7bc5cf5c-gwtrd" Jul 13 14:22:23.319846 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:23.319807 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:23.384953 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:23.384924 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:25.981168 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:25.981102 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5647674757-gwgns" podUID="b9f13eee-31c5-40f5-a420-c1214e371eb2" containerName="registry" containerID="cri-o://eed16fcd5aea99755627686e31617c010bd4402f2e435c7d2b3c38a8d2f2abd1" gracePeriod=30 Jul 13 14:22:26.232937 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.232854 2566 generic.go:356] "Generic (PLEG): container finished" podID="b9f13eee-31c5-40f5-a420-c1214e371eb2" containerID="eed16fcd5aea99755627686e31617c010bd4402f2e435c7d2b3c38a8d2f2abd1" exitCode=0 Jul 13 14:22:26.232937 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.232914 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5647674757-gwgns" event={"ID":"b9f13eee-31c5-40f5-a420-c1214e371eb2","Type":"ContainerDied","Data":"eed16fcd5aea99755627686e31617c010bd4402f2e435c7d2b3c38a8d2f2abd1"} Jul 13 14:22:26.272576 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.272550 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:22:26.350834 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.350777 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:26.435939 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.435914 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls\") pod \"b9f13eee-31c5-40f5-a420-c1214e371eb2\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " Jul 13 14:22:26.436120 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.435945 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9f13eee-31c5-40f5-a420-c1214e371eb2-image-registry-private-configuration\") pod \"b9f13eee-31c5-40f5-a420-c1214e371eb2\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " Jul 13 14:22:26.436120 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.435975 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztqvm\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-kube-api-access-ztqvm\") pod \"b9f13eee-31c5-40f5-a420-c1214e371eb2\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " Jul 13 14:22:26.436120 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.435998 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-bound-sa-token\") pod \"b9f13eee-31c5-40f5-a420-c1214e371eb2\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " Jul 13 14:22:26.436120 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.436017 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9f13eee-31c5-40f5-a420-c1214e371eb2-trusted-ca\") pod \"b9f13eee-31c5-40f5-a420-c1214e371eb2\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " Jul 13 14:22:26.436120 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.436083 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-certificates\") pod \"b9f13eee-31c5-40f5-a420-c1214e371eb2\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " Jul 13 14:22:26.436120 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.436101 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9f13eee-31c5-40f5-a420-c1214e371eb2-installation-pull-secrets\") pod \"b9f13eee-31c5-40f5-a420-c1214e371eb2\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " Jul 13 14:22:26.436416 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.436131 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9f13eee-31c5-40f5-a420-c1214e371eb2-ca-trust-extracted\") pod \"b9f13eee-31c5-40f5-a420-c1214e371eb2\" (UID: \"b9f13eee-31c5-40f5-a420-c1214e371eb2\") " Jul 13 14:22:26.436694 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.436575 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b9f13eee-31c5-40f5-a420-c1214e371eb2" (UID: "b9f13eee-31c5-40f5-a420-c1214e371eb2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:22:26.436890 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.436819 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f13eee-31c5-40f5-a420-c1214e371eb2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b9f13eee-31c5-40f5-a420-c1214e371eb2" (UID: "b9f13eee-31c5-40f5-a420-c1214e371eb2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:22:26.438872 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.438838 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b9f13eee-31c5-40f5-a420-c1214e371eb2" (UID: "b9f13eee-31c5-40f5-a420-c1214e371eb2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:22:26.438981 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.438945 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-kube-api-access-ztqvm" (OuterVolumeSpecName: "kube-api-access-ztqvm") pod "b9f13eee-31c5-40f5-a420-c1214e371eb2" (UID: "b9f13eee-31c5-40f5-a420-c1214e371eb2"). InnerVolumeSpecName "kube-api-access-ztqvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:22:26.439045 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.439029 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f13eee-31c5-40f5-a420-c1214e371eb2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b9f13eee-31c5-40f5-a420-c1214e371eb2" (UID: "b9f13eee-31c5-40f5-a420-c1214e371eb2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:22:26.439142 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.439112 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f13eee-31c5-40f5-a420-c1214e371eb2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b9f13eee-31c5-40f5-a420-c1214e371eb2" (UID: "b9f13eee-31c5-40f5-a420-c1214e371eb2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:22:26.439203 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.439164 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b9f13eee-31c5-40f5-a420-c1214e371eb2" (UID: "b9f13eee-31c5-40f5-a420-c1214e371eb2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:22:26.444627 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.444602 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f13eee-31c5-40f5-a420-c1214e371eb2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b9f13eee-31c5-40f5-a420-c1214e371eb2" (UID: "b9f13eee-31c5-40f5-a420-c1214e371eb2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:22:26.448251 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.448197 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:26.501124 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.501049 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6z662"] Jul 13 14:22:26.501293 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.501280 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f13eee-31c5-40f5-a420-c1214e371eb2" containerName="registry" Jul 13 14:22:26.501333 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.501294 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f13eee-31c5-40f5-a420-c1214e371eb2" containerName="registry" Jul 13 14:22:26.501365 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.501346 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9f13eee-31c5-40f5-a420-c1214e371eb2" containerName="registry" Jul 13 14:22:26.519524 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.519495 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:26.537563 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.537539 2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-certificates\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:22:26.537563 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.537558 2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9f13eee-31c5-40f5-a420-c1214e371eb2-installation-pull-secrets\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:22:26.537737 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.537570 2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9f13eee-31c5-40f5-a420-c1214e371eb2-ca-trust-extracted\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:22:26.537737 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.537579 2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-registry-tls\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:22:26.537737 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.537588 2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9f13eee-31c5-40f5-a420-c1214e371eb2-image-registry-private-configuration\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:22:26.537737 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.537598 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztqvm\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-kube-api-access-ztqvm\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:22:26.537737 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.537607 2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9f13eee-31c5-40f5-a420-c1214e371eb2-bound-sa-token\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:22:26.537737 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.537615 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9f13eee-31c5-40f5-a420-c1214e371eb2-trusted-ca\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:22:26.538332 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.538317 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.540848 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.540818 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 13 14:22:26.540961 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.540919 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 13 14:22:26.541031 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.541019 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 13 14:22:26.541131 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.541111 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 13 14:22:26.541242 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.541137 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 13 14:22:26.541332 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.541317 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 13 14:22:26.541395 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.541381 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p5shv\"" Jul 13 14:22:26.593055 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.593027 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:26.638218 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.638191 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-wtmp\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.638370 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.638230 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzn7c\" (UniqueName: \"kubernetes.io/projected/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-kube-api-access-gzn7c\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.638370 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.638249 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-sys\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.638370 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.638304 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.638370 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.638347 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-accelerators-collector-config\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.638579 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.638394 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-metrics-client-ca\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.638579 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.638450 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-textfile\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.638579 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.638483 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-tls\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.638579 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.638509 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-root\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.693905 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.693878 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:26.739723 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.739685 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzn7c\" (UniqueName: \"kubernetes.io/projected/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-kube-api-access-gzn7c\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.739723 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.739726 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-sys\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.739956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.739750 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.739956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.739775 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-accelerators-collector-config\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.739956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.739820 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-sys\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.739956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.739835 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-metrics-client-ca\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.739956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.739899 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-textfile\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.739956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.739934 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-tls\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.739956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.739955 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-root\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.740302 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.739997 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-wtmp\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.740302 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:22:26.740104 2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 13 14:22:26.740302 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.740149 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-wtmp\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.740302 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:22:26.740172 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-tls podName:1a3ae5cf-ea86-49cd-a760-ce305f3bb203 nodeName:}" failed. No retries permitted until 2026-07-13 14:22:27.240153086 +0000 UTC m=+88.924573383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-tls") pod "node-exporter-6z662" (UID: "1a3ae5cf-ea86-49cd-a760-ce305f3bb203") : secret "node-exporter-tls" not found Jul 13 14:22:26.740302 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.740183 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-root\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.740563 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.740351 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-textfile\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.740563 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.740452 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-accelerators-collector-config\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.740563 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.740535 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-metrics-client-ca\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.742194 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.742169 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.750691 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.750670 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzn7c\" (UniqueName: \"kubernetes.io/projected/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-kube-api-access-gzn7c\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:26.794205 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.794146 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:26.894072 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:26.894041 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:27.237815 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:27.237769 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5647674757-gwgns" event={"ID":"b9f13eee-31c5-40f5-a420-c1214e371eb2","Type":"ContainerDied","Data":"b972e54af37143318a70ab40b79a3a3d468437eb740786e026f6337236ebb700"} Jul 13 14:22:27.238193 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:27.237823 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5647674757-gwgns" Jul 13 14:22:27.238193 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:27.237876 2566 scope.go:117] "RemoveContainer" containerID="eed16fcd5aea99755627686e31617c010bd4402f2e435c7d2b3c38a8d2f2abd1" Jul 13 14:22:27.246592 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:27.246548 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-tls\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:27.248648 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:27.248619 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a3ae5cf-ea86-49cd-a760-ce305f3bb203-node-exporter-tls\") pod \"node-exporter-6z662\" (UID: \"1a3ae5cf-ea86-49cd-a760-ce305f3bb203\") " pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:27.258296 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:27.258270 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5647674757-gwgns"] Jul 13 14:22:27.262453 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:27.262432 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5647674757-gwgns"] Jul 13 14:22:27.431805 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:27.431754 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:27.447821 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:27.447780 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6z662" Jul 13 14:22:27.457596 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:22:27.457572 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3ae5cf_ea86_49cd_a760_ce305f3bb203.slice/crio-97ace77dd3b455dfc7688d81254baa1dd5ea4b4ee8fc8aacaffebe66dd446790 WatchSource:0}: Error finding container 97ace77dd3b455dfc7688d81254baa1dd5ea4b4ee8fc8aacaffebe66dd446790: Status 404 returned error can't find the container with id 97ace77dd3b455dfc7688d81254baa1dd5ea4b4ee8fc8aacaffebe66dd446790 Jul 13 14:22:27.570506 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:27.570484 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:27.652835 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:27.652806 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:28.195067 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:28.195036 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:28.242279 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:28.242243 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6z662" event={"ID":"1a3ae5cf-ea86-49cd-a760-ce305f3bb203","Type":"ContainerStarted","Data":"97ace77dd3b455dfc7688d81254baa1dd5ea4b4ee8fc8aacaffebe66dd446790"} Jul 13 14:22:28.866139 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:28.866106 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9f13eee-31c5-40f5-a420-c1214e371eb2" path="/var/lib/kubelet/pods/b9f13eee-31c5-40f5-a420-c1214e371eb2/volumes" Jul 13 14:22:29.248063 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:29.247971 2566 generic.go:356] "Generic (PLEG): container finished" podID="1a3ae5cf-ea86-49cd-a760-ce305f3bb203" containerID="8e11b4ca6eff77a4c3fdb2dd64802c5ad0629e922e47937582d827490d7745cc" exitCode=0 Jul 13 14:22:29.248063 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:29.248043 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6z662" event={"ID":"1a3ae5cf-ea86-49cd-a760-ce305f3bb203","Type":"ContainerDied","Data":"8e11b4ca6eff77a4c3fdb2dd64802c5ad0629e922e47937582d827490d7745cc"} Jul 13 14:22:29.289880 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:29.289853 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:29.396562 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:29.396533 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:29.491364 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:29.491338 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:29.852503 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:29.852471 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:30.161994 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:30.161910 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:30.240074 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:30.240044 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:30.253597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:30.253571 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6z662" event={"ID":"1a3ae5cf-ea86-49cd-a760-ce305f3bb203","Type":"ContainerStarted","Data":"917f7fedf40aaa0b9674db43fe58b22ee400e9ffead7d02eed1a4cf1656069ce"} Jul 13 14:22:30.253933 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:30.253616 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6z662" event={"ID":"1a3ae5cf-ea86-49cd-a760-ce305f3bb203","Type":"ContainerStarted","Data":"1b8917a9cda1fca192f2e255909464cdcf3df19fa741d25e13013c0c9d3ef66a"} Jul 13 14:22:30.277226 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:30.277188 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6z662" podStartSLOduration=3.298944907 podStartE2EDuration="4.277175312s" totalImagesPullingTime="915.817708ms" totalInitContainerRuntime="62.412697ms" isStatefulPod=true podCreationTimestamp="2026-07-13 14:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:22:30.276136366 +0000 UTC m=+91.960556672" watchObservedRunningTime="2026-07-13 14:22:30.277175312 +0000 UTC m=+91.961595618" Jul 13 14:22:30.426142 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:30.426067 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:31.284291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:31.284247 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:31.368774 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:31.368742 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:31.436479 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:31.436448 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:31.515952 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:31.515927 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:32.788889 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:32.788859 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:32.860240 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:32.860207 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:35.771754 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:35.771722 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:35.836863 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:35.836834 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:38.211519 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:38.211474 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f9447d85f-z292b" podUID="4889381b-b506-4b1a-8af5-9a10edb4ce98" containerName="console" containerID="cri-o://02b211f8932324c02b2ed1cd828d42f04182357d3ecb4aab46eb6dbc6a8e68ba" gracePeriod=15 Jul 13 14:22:38.980605 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:38.980581 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:39.030873 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.030844 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-oauth-config\") pod \"4889381b-b506-4b1a-8af5-9a10edb4ce98\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " Jul 13 14:22:39.031015 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.030879 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-service-ca\") pod \"4889381b-b506-4b1a-8af5-9a10edb4ce98\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " Jul 13 14:22:39.031015 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.030898 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-serving-cert\") pod \"4889381b-b506-4b1a-8af5-9a10edb4ce98\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " Jul 13 14:22:39.031015 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.030935 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8ccx\" (UniqueName: \"kubernetes.io/projected/4889381b-b506-4b1a-8af5-9a10edb4ce98-kube-api-access-p8ccx\") pod \"4889381b-b506-4b1a-8af5-9a10edb4ce98\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " Jul 13 14:22:39.031015 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.030968 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-oauth-serving-cert\") pod \"4889381b-b506-4b1a-8af5-9a10edb4ce98\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " Jul 13 14:22:39.031015 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.031006 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-config\") pod \"4889381b-b506-4b1a-8af5-9a10edb4ce98\" (UID: \"4889381b-b506-4b1a-8af5-9a10edb4ce98\") " Jul 13 14:22:39.031328 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.031301 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-service-ca" (OuterVolumeSpecName: "service-ca") pod "4889381b-b506-4b1a-8af5-9a10edb4ce98" (UID: "4889381b-b506-4b1a-8af5-9a10edb4ce98"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:22:39.031459 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.031375 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4889381b-b506-4b1a-8af5-9a10edb4ce98" (UID: "4889381b-b506-4b1a-8af5-9a10edb4ce98"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:22:39.031459 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.031443 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-config" (OuterVolumeSpecName: "console-config") pod "4889381b-b506-4b1a-8af5-9a10edb4ce98" (UID: "4889381b-b506-4b1a-8af5-9a10edb4ce98"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:22:39.033224 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.033202 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4889381b-b506-4b1a-8af5-9a10edb4ce98" (UID: "4889381b-b506-4b1a-8af5-9a10edb4ce98"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:22:39.033340 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.033319 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4889381b-b506-4b1a-8af5-9a10edb4ce98-kube-api-access-p8ccx" (OuterVolumeSpecName: "kube-api-access-p8ccx") pod "4889381b-b506-4b1a-8af5-9a10edb4ce98" (UID: "4889381b-b506-4b1a-8af5-9a10edb4ce98"). InnerVolumeSpecName "kube-api-access-p8ccx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:22:39.033381 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.033331 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4889381b-b506-4b1a-8af5-9a10edb4ce98" (UID: "4889381b-b506-4b1a-8af5-9a10edb4ce98"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:22:39.131763 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.131703 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-config\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:22:39.131763 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.131726 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-oauth-config\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:22:39.131763 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.131736 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-service-ca\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:22:39.131763 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.131745 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4889381b-b506-4b1a-8af5-9a10edb4ce98-console-serving-cert\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:22:39.131763 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.131754 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8ccx\" (UniqueName: \"kubernetes.io/projected/4889381b-b506-4b1a-8af5-9a10edb4ce98-kube-api-access-p8ccx\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:22:39.131763 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.131764 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4889381b-b506-4b1a-8af5-9a10edb4ce98-oauth-serving-cert\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:22:39.284614 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.284579 2566 generic.go:356] "Generic (PLEG): container finished" podID="4889381b-b506-4b1a-8af5-9a10edb4ce98" containerID="02b211f8932324c02b2ed1cd828d42f04182357d3ecb4aab46eb6dbc6a8e68ba" exitCode=0 Jul 13 14:22:39.285030 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.284625 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9447d85f-z292b" Jul 13 14:22:39.285030 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.284636 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9447d85f-z292b" event={"ID":"4889381b-b506-4b1a-8af5-9a10edb4ce98","Type":"ContainerDied","Data":"02b211f8932324c02b2ed1cd828d42f04182357d3ecb4aab46eb6dbc6a8e68ba"} Jul 13 14:22:39.285030 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.284674 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9447d85f-z292b" event={"ID":"4889381b-b506-4b1a-8af5-9a10edb4ce98","Type":"ContainerDied","Data":"ba9c6fcc091eb36ca887eadfbddbaa12685e861d861e32417fcf553558f5873c"} Jul 13 14:22:39.285030 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.284694 2566 scope.go:117] "RemoveContainer" containerID="02b211f8932324c02b2ed1cd828d42f04182357d3ecb4aab46eb6dbc6a8e68ba" Jul 13 14:22:39.293079 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.293063 2566 scope.go:117] "RemoveContainer" containerID="02b211f8932324c02b2ed1cd828d42f04182357d3ecb4aab46eb6dbc6a8e68ba" Jul 13 14:22:39.293353 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:22:39.293331 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02b211f8932324c02b2ed1cd828d42f04182357d3ecb4aab46eb6dbc6a8e68ba\": container with ID starting with 02b211f8932324c02b2ed1cd828d42f04182357d3ecb4aab46eb6dbc6a8e68ba not found: ID does not exist" containerID="02b211f8932324c02b2ed1cd828d42f04182357d3ecb4aab46eb6dbc6a8e68ba" Jul 13 14:22:39.293408 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.293364 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02b211f8932324c02b2ed1cd828d42f04182357d3ecb4aab46eb6dbc6a8e68ba"} err="failed to get container status \"02b211f8932324c02b2ed1cd828d42f04182357d3ecb4aab46eb6dbc6a8e68ba\": rpc error: code = NotFound desc = could not find container \"02b211f8932324c02b2ed1cd828d42f04182357d3ecb4aab46eb6dbc6a8e68ba\": container with ID starting with 02b211f8932324c02b2ed1cd828d42f04182357d3ecb4aab46eb6dbc6a8e68ba not found: ID does not exist" Jul 13 14:22:39.307827 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.307807 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f9447d85f-z292b"] Jul 13 14:22:39.313440 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:39.313422 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f9447d85f-z292b"] Jul 13 14:22:40.866233 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:40.866198 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4889381b-b506-4b1a-8af5-9a10edb4ce98" path="/var/lib/kubelet/pods/4889381b-b506-4b1a-8af5-9a10edb4ce98/volumes" Jul 13 14:22:42.186837 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:42.186779 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sl54c" Jul 13 14:22:48.001681 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:48.001613 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:48.102163 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:48.102132 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:48.161803 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:48.161760 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:50.776090 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:50.776053 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:22:50.838595 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:22:50.838565 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:23:03.832197 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:23:03.832161 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:23:42.597076 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:23:42.597041 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:23:42.683929 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:23:42.683882 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:23:46.603010 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:23:46.602981 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:23:54.456348 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:23:54.456273 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:23:54.556926 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:23:54.556887 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:23:57.269290 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:23:57.269216 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:23:57.368938 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:23:57.368890 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:23:57.432824 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:23:57.432799 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:24:05.776123 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:24:05.776088 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:24:05.882813 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:24:05.882757 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:24:05.987625 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:24:05.987595 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:24:06.436931 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:24:06.436895 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:24:43.220562 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:24:43.220536 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:01.567853 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:01.567813 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:01.756045 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:01.756010 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:02.195324 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.195291 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl"] Jul 13 14:25:02.196064 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.196039 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4889381b-b506-4b1a-8af5-9a10edb4ce98" containerName="console" Jul 13 14:25:02.196213 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.196202 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4889381b-b506-4b1a-8af5-9a10edb4ce98" containerName="console" Jul 13 14:25:02.196408 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.196397 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4889381b-b506-4b1a-8af5-9a10edb4ce98" containerName="console" Jul 13 14:25:02.200328 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.200310 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" Jul 13 14:25:02.203868 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.203849 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 13 14:25:02.204202 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.204177 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 13 14:25:02.204528 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.204515 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fvfkz\"" Jul 13 14:25:02.209980 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.209958 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl"] Jul 13 14:25:02.289043 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.289015 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67fb5f22-1a7d-4201-8066-e86af046df7d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl\" (UID: \"67fb5f22-1a7d-4201-8066-e86af046df7d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" Jul 13 14:25:02.289173 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.289055 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2crtw\" (UniqueName: \"kubernetes.io/projected/67fb5f22-1a7d-4201-8066-e86af046df7d-kube-api-access-2crtw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl\" (UID: \"67fb5f22-1a7d-4201-8066-e86af046df7d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" Jul 13 14:25:02.289173 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.289085 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67fb5f22-1a7d-4201-8066-e86af046df7d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl\" (UID: \"67fb5f22-1a7d-4201-8066-e86af046df7d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" Jul 13 14:25:02.390362 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.390331 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2crtw\" (UniqueName: \"kubernetes.io/projected/67fb5f22-1a7d-4201-8066-e86af046df7d-kube-api-access-2crtw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl\" (UID: \"67fb5f22-1a7d-4201-8066-e86af046df7d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" Jul 13 14:25:02.390482 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.390380 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67fb5f22-1a7d-4201-8066-e86af046df7d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl\" (UID: \"67fb5f22-1a7d-4201-8066-e86af046df7d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" Jul 13 14:25:02.390482 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.390413 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67fb5f22-1a7d-4201-8066-e86af046df7d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl\" (UID: \"67fb5f22-1a7d-4201-8066-e86af046df7d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" Jul 13 14:25:02.390820 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.390800 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67fb5f22-1a7d-4201-8066-e86af046df7d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl\" (UID: \"67fb5f22-1a7d-4201-8066-e86af046df7d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" Jul 13 14:25:02.390862 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.390815 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67fb5f22-1a7d-4201-8066-e86af046df7d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl\" (UID: \"67fb5f22-1a7d-4201-8066-e86af046df7d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" Jul 13 14:25:02.400025 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.399999 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2crtw\" (UniqueName: \"kubernetes.io/projected/67fb5f22-1a7d-4201-8066-e86af046df7d-kube-api-access-2crtw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl\" (UID: \"67fb5f22-1a7d-4201-8066-e86af046df7d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" Jul 13 14:25:02.508922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.508846 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" Jul 13 14:25:02.643436 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.643411 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl"] Jul 13 14:25:02.645697 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:25:02.645667 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fb5f22_1a7d_4201_8066_e86af046df7d.slice/crio-70440335bcc46e20a2527e997bcbcb6e86d7dc0619074b4d53e0bd330cfc830c WatchSource:0}: Error finding container 70440335bcc46e20a2527e997bcbcb6e86d7dc0619074b4d53e0bd330cfc830c: Status 404 returned error can't find the container with id 70440335bcc46e20a2527e997bcbcb6e86d7dc0619074b4d53e0bd330cfc830c Jul 13 14:25:02.745264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:02.745232 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" event={"ID":"67fb5f22-1a7d-4201-8066-e86af046df7d","Type":"ContainerStarted","Data":"70440335bcc46e20a2527e997bcbcb6e86d7dc0619074b4d53e0bd330cfc830c"} Jul 13 14:25:07.763965 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:07.763922 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" event={"ID":"67fb5f22-1a7d-4201-8066-e86af046df7d","Type":"ContainerStarted","Data":"a832b7289297856cad99a398edfde75cb3f2d1d1f2112386ce8cd6bcf372a9be"} Jul 13 14:25:08.767966 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:08.767925 2566 generic.go:356] "Generic (PLEG): container finished" podID="67fb5f22-1a7d-4201-8066-e86af046df7d" containerID="a832b7289297856cad99a398edfde75cb3f2d1d1f2112386ce8cd6bcf372a9be" exitCode=0 Jul 13 14:25:08.768336 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:08.768010 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" event={"ID":"67fb5f22-1a7d-4201-8066-e86af046df7d","Type":"ContainerDied","Data":"a832b7289297856cad99a398edfde75cb3f2d1d1f2112386ce8cd6bcf372a9be"} Jul 13 14:25:12.786202 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:12.786168 2566 generic.go:356] "Generic (PLEG): container finished" podID="67fb5f22-1a7d-4201-8066-e86af046df7d" containerID="8f428d828d25e12acb5f0ef33ba394ef959ae8725cd5ceafbb0fe34e813a9b9d" exitCode=0 Jul 13 14:25:12.786664 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:12.786220 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" event={"ID":"67fb5f22-1a7d-4201-8066-e86af046df7d","Type":"ContainerDied","Data":"8f428d828d25e12acb5f0ef33ba394ef959ae8725cd5ceafbb0fe34e813a9b9d"} Jul 13 14:25:17.199887 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:17.199852 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:20.816750 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:20.816714 2566 generic.go:356] "Generic (PLEG): container finished" podID="67fb5f22-1a7d-4201-8066-e86af046df7d" containerID="3bf5feae197477de46a132b86b8e270141fd58977908354fa4d2052914187730" exitCode=0 Jul 13 14:25:20.817116 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:20.816758 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" event={"ID":"67fb5f22-1a7d-4201-8066-e86af046df7d","Type":"ContainerDied","Data":"3bf5feae197477de46a132b86b8e270141fd58977908354fa4d2052914187730"} Jul 13 14:25:21.940301 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:21.940280 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" Jul 13 14:25:22.034802 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:22.034750 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67fb5f22-1a7d-4201-8066-e86af046df7d-util\") pod \"67fb5f22-1a7d-4201-8066-e86af046df7d\" (UID: \"67fb5f22-1a7d-4201-8066-e86af046df7d\") " Jul 13 14:25:22.034952 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:22.034816 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2crtw\" (UniqueName: \"kubernetes.io/projected/67fb5f22-1a7d-4201-8066-e86af046df7d-kube-api-access-2crtw\") pod \"67fb5f22-1a7d-4201-8066-e86af046df7d\" (UID: \"67fb5f22-1a7d-4201-8066-e86af046df7d\") " Jul 13 14:25:22.034952 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:22.034862 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67fb5f22-1a7d-4201-8066-e86af046df7d-bundle\") pod \"67fb5f22-1a7d-4201-8066-e86af046df7d\" (UID: \"67fb5f22-1a7d-4201-8066-e86af046df7d\") " Jul 13 14:25:22.035456 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:22.035430 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67fb5f22-1a7d-4201-8066-e86af046df7d-bundle" (OuterVolumeSpecName: "bundle") pod "67fb5f22-1a7d-4201-8066-e86af046df7d" (UID: "67fb5f22-1a7d-4201-8066-e86af046df7d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:25:22.036885 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:22.036850 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fb5f22-1a7d-4201-8066-e86af046df7d-kube-api-access-2crtw" (OuterVolumeSpecName: "kube-api-access-2crtw") pod "67fb5f22-1a7d-4201-8066-e86af046df7d" (UID: "67fb5f22-1a7d-4201-8066-e86af046df7d"). InnerVolumeSpecName "kube-api-access-2crtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:25:22.040139 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:22.040117 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67fb5f22-1a7d-4201-8066-e86af046df7d-util" (OuterVolumeSpecName: "util") pod "67fb5f22-1a7d-4201-8066-e86af046df7d" (UID: "67fb5f22-1a7d-4201-8066-e86af046df7d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:25:22.136391 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:22.136298 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67fb5f22-1a7d-4201-8066-e86af046df7d-util\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:25:22.136391 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:22.136339 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2crtw\" (UniqueName: \"kubernetes.io/projected/67fb5f22-1a7d-4201-8066-e86af046df7d-kube-api-access-2crtw\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:25:22.136391 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:22.136349 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67fb5f22-1a7d-4201-8066-e86af046df7d-bundle\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:25:22.826427 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:22.826399 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" Jul 13 14:25:22.826584 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:22.826403 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406mdzl" event={"ID":"67fb5f22-1a7d-4201-8066-e86af046df7d","Type":"ContainerDied","Data":"70440335bcc46e20a2527e997bcbcb6e86d7dc0619074b4d53e0bd330cfc830c"} Jul 13 14:25:22.826584 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:22.826523 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70440335bcc46e20a2527e997bcbcb6e86d7dc0619074b4d53e0bd330cfc830c" Jul 13 14:25:28.512617 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:28.512583 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:28.896796 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:28.896754 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:32.794134 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:32.794106 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:32.877439 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:32.877409 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:34.613545 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:34.613511 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:34.706546 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:34.706510 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:35.147187 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.147155 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:35.318009 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.317961 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:35.476350 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.476275 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v"] Jul 13 14:25:35.476506 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.476495 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67fb5f22-1a7d-4201-8066-e86af046df7d" containerName="extract" Jul 13 14:25:35.476543 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.476507 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="67fb5f22-1a7d-4201-8066-e86af046df7d" containerName="extract" Jul 13 14:25:35.476543 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.476519 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67fb5f22-1a7d-4201-8066-e86af046df7d" containerName="pull" Jul 13 14:25:35.476543 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.476524 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="67fb5f22-1a7d-4201-8066-e86af046df7d" containerName="pull" Jul 13 14:25:35.476543 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.476538 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67fb5f22-1a7d-4201-8066-e86af046df7d" containerName="util" Jul 13 14:25:35.476543 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.476543 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="67fb5f22-1a7d-4201-8066-e86af046df7d" containerName="util" Jul 13 14:25:35.476687 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.476584 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="67fb5f22-1a7d-4201-8066-e86af046df7d" containerName="extract" Jul 13 14:25:35.481129 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.481112 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" Jul 13 14:25:35.484221 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.484191 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 13 14:25:35.486870 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.486851 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fvfkz\"" Jul 13 14:25:35.486870 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.486864 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 13 14:25:35.495611 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.495590 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v"] Jul 13 14:25:35.627846 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.627813 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wknk8\" (UniqueName: \"kubernetes.io/projected/019bd724-abc9-4054-8171-b68e4533ccd9-kube-api-access-wknk8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v\" (UID: \"019bd724-abc9-4054-8171-b68e4533ccd9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" Jul 13 14:25:35.627846 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.627848 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/019bd724-abc9-4054-8171-b68e4533ccd9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v\" (UID: \"019bd724-abc9-4054-8171-b68e4533ccd9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" Jul 13 14:25:35.628237 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.627879 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/019bd724-abc9-4054-8171-b68e4533ccd9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v\" (UID: \"019bd724-abc9-4054-8171-b68e4533ccd9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" Jul 13 14:25:35.728731 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.728655 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wknk8\" (UniqueName: \"kubernetes.io/projected/019bd724-abc9-4054-8171-b68e4533ccd9-kube-api-access-wknk8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v\" (UID: \"019bd724-abc9-4054-8171-b68e4533ccd9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" Jul 13 14:25:35.728731 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.728687 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/019bd724-abc9-4054-8171-b68e4533ccd9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v\" (UID: \"019bd724-abc9-4054-8171-b68e4533ccd9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" Jul 13 14:25:35.728731 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.728715 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/019bd724-abc9-4054-8171-b68e4533ccd9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v\" (UID: \"019bd724-abc9-4054-8171-b68e4533ccd9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" Jul 13 14:25:35.729149 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.729130 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/019bd724-abc9-4054-8171-b68e4533ccd9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v\" (UID: \"019bd724-abc9-4054-8171-b68e4533ccd9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" Jul 13 14:25:35.729188 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.729133 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/019bd724-abc9-4054-8171-b68e4533ccd9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v\" (UID: \"019bd724-abc9-4054-8171-b68e4533ccd9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" Jul 13 14:25:35.776931 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.776895 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wknk8\" (UniqueName: \"kubernetes.io/projected/019bd724-abc9-4054-8171-b68e4533ccd9-kube-api-access-wknk8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v\" (UID: \"019bd724-abc9-4054-8171-b68e4533ccd9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" Jul 13 14:25:35.790705 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.790686 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" Jul 13 14:25:35.933297 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:35.933269 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v"] Jul 13 14:25:35.936649 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:25:35.936624 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019bd724_abc9_4054_8171_b68e4533ccd9.slice/crio-dcfe739679d9773fb9f2116723ac37c0ac390db749768450b984a36a9dd12ce2 WatchSource:0}: Error finding container dcfe739679d9773fb9f2116723ac37c0ac390db749768450b984a36a9dd12ce2: Status 404 returned error can't find the container with id dcfe739679d9773fb9f2116723ac37c0ac390db749768450b984a36a9dd12ce2 Jul 13 14:25:36.872627 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:36.872598 2566 generic.go:356] "Generic (PLEG): container finished" podID="019bd724-abc9-4054-8171-b68e4533ccd9" containerID="e695b491096dfbebe9351a4bae5e334c4a0f06a0eee16350c96d65e18ad59db1" exitCode=0 Jul 13 14:25:36.872984 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:36.872654 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" event={"ID":"019bd724-abc9-4054-8171-b68e4533ccd9","Type":"ContainerDied","Data":"e695b491096dfbebe9351a4bae5e334c4a0f06a0eee16350c96d65e18ad59db1"} Jul 13 14:25:36.872984 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:36.872689 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" event={"ID":"019bd724-abc9-4054-8171-b68e4533ccd9","Type":"ContainerStarted","Data":"dcfe739679d9773fb9f2116723ac37c0ac390db749768450b984a36a9dd12ce2"} Jul 13 14:25:37.107199 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:37.107170 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:37.294560 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:37.294531 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:38.881982 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:38.881940 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" event={"ID":"019bd724-abc9-4054-8171-b68e4533ccd9","Type":"ContainerStarted","Data":"bbc9b4327bb163174b09ae562be04a4491ddf14e79451967bd05961d163cc212"} Jul 13 14:25:39.170417 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:39.170352 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:39.307334 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:39.307296 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:39.886914 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:39.886874 2566 generic.go:356] "Generic (PLEG): container finished" podID="019bd724-abc9-4054-8171-b68e4533ccd9" containerID="bbc9b4327bb163174b09ae562be04a4491ddf14e79451967bd05961d163cc212" exitCode=0 Jul 13 14:25:39.887298 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:39.886963 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" event={"ID":"019bd724-abc9-4054-8171-b68e4533ccd9","Type":"ContainerDied","Data":"bbc9b4327bb163174b09ae562be04a4491ddf14e79451967bd05961d163cc212"} Jul 13 14:25:40.892223 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:40.892189 2566 generic.go:356] "Generic (PLEG): container finished" podID="019bd724-abc9-4054-8171-b68e4533ccd9" containerID="4c9169081e0e5cb601b35d678d2f849d1e8e3e7f09917855af9f620c986f3b63" exitCode=0 Jul 13 14:25:40.892605 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:40.892260 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" event={"ID":"019bd724-abc9-4054-8171-b68e4533ccd9","Type":"ContainerDied","Data":"4c9169081e0e5cb601b35d678d2f849d1e8e3e7f09917855af9f620c986f3b63"} Jul 13 14:25:42.018925 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:42.018902 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" Jul 13 14:25:42.175151 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:42.175077 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wknk8\" (UniqueName: \"kubernetes.io/projected/019bd724-abc9-4054-8171-b68e4533ccd9-kube-api-access-wknk8\") pod \"019bd724-abc9-4054-8171-b68e4533ccd9\" (UID: \"019bd724-abc9-4054-8171-b68e4533ccd9\") " Jul 13 14:25:42.175151 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:42.175126 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/019bd724-abc9-4054-8171-b68e4533ccd9-bundle\") pod \"019bd724-abc9-4054-8171-b68e4533ccd9\" (UID: \"019bd724-abc9-4054-8171-b68e4533ccd9\") " Jul 13 14:25:42.175151 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:42.175142 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/019bd724-abc9-4054-8171-b68e4533ccd9-util\") pod \"019bd724-abc9-4054-8171-b68e4533ccd9\" (UID: \"019bd724-abc9-4054-8171-b68e4533ccd9\") " Jul 13 14:25:42.175504 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:42.175477 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019bd724-abc9-4054-8171-b68e4533ccd9-bundle" (OuterVolumeSpecName: "bundle") pod "019bd724-abc9-4054-8171-b68e4533ccd9" (UID: "019bd724-abc9-4054-8171-b68e4533ccd9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:25:42.177136 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:42.177116 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/019bd724-abc9-4054-8171-b68e4533ccd9-kube-api-access-wknk8" (OuterVolumeSpecName: "kube-api-access-wknk8") pod "019bd724-abc9-4054-8171-b68e4533ccd9" (UID: "019bd724-abc9-4054-8171-b68e4533ccd9"). InnerVolumeSpecName "kube-api-access-wknk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:25:42.180863 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:42.180840 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019bd724-abc9-4054-8171-b68e4533ccd9-util" (OuterVolumeSpecName: "util") pod "019bd724-abc9-4054-8171-b68e4533ccd9" (UID: "019bd724-abc9-4054-8171-b68e4533ccd9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:25:42.275919 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:42.275890 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wknk8\" (UniqueName: \"kubernetes.io/projected/019bd724-abc9-4054-8171-b68e4533ccd9-kube-api-access-wknk8\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:25:42.275919 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:42.275913 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/019bd724-abc9-4054-8171-b68e4533ccd9-bundle\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:25:42.275919 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:42.275925 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/019bd724-abc9-4054-8171-b68e4533ccd9-util\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:25:42.901043 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:42.901009 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" event={"ID":"019bd724-abc9-4054-8171-b68e4533ccd9","Type":"ContainerDied","Data":"dcfe739679d9773fb9f2116723ac37c0ac390db749768450b984a36a9dd12ce2"} Jul 13 14:25:42.901043 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:42.901034 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f44j2v" Jul 13 14:25:42.901201 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:42.901063 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcfe739679d9773fb9f2116723ac37c0ac390db749768450b984a36a9dd12ce2" Jul 13 14:25:44.429148 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:44.429116 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:46.452317 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:46.452282 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:46.561903 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:46.561868 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:52.435345 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:52.435314 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:52.565040 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:52.565012 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:54.343616 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.343585 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq"] Jul 13 14:25:54.343985 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.343829 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="019bd724-abc9-4054-8171-b68e4533ccd9" containerName="extract" Jul 13 14:25:54.343985 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.343840 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="019bd724-abc9-4054-8171-b68e4533ccd9" containerName="extract" Jul 13 14:25:54.343985 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.343849 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="019bd724-abc9-4054-8171-b68e4533ccd9" containerName="util" Jul 13 14:25:54.343985 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.343855 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="019bd724-abc9-4054-8171-b68e4533ccd9" containerName="util" Jul 13 14:25:54.343985 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.343867 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="019bd724-abc9-4054-8171-b68e4533ccd9" containerName="pull" Jul 13 14:25:54.343985 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.343873 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="019bd724-abc9-4054-8171-b68e4533ccd9" containerName="pull" Jul 13 14:25:54.343985 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.343913 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="019bd724-abc9-4054-8171-b68e4533ccd9" containerName="extract" Jul 13 14:25:54.346071 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.346051 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" Jul 13 14:25:54.349344 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.349325 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fvfkz\"" Jul 13 14:25:54.350315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.350297 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 13 14:25:54.350434 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.350349 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 13 14:25:54.361138 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.361117 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq"] Jul 13 14:25:54.464803 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.464756 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af0c7848-8392-473a-8c56-4e5d50e2f099-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq\" (UID: \"af0c7848-8392-473a-8c56-4e5d50e2f099\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" Jul 13 14:25:54.464964 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.464837 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nmfv\" (UniqueName: \"kubernetes.io/projected/af0c7848-8392-473a-8c56-4e5d50e2f099-kube-api-access-4nmfv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq\" (UID: \"af0c7848-8392-473a-8c56-4e5d50e2f099\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" Jul 13 14:25:54.464964 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.464879 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af0c7848-8392-473a-8c56-4e5d50e2f099-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq\" (UID: \"af0c7848-8392-473a-8c56-4e5d50e2f099\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" Jul 13 14:25:54.566057 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.566020 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af0c7848-8392-473a-8c56-4e5d50e2f099-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq\" (UID: \"af0c7848-8392-473a-8c56-4e5d50e2f099\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" Jul 13 14:25:54.566170 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.566073 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af0c7848-8392-473a-8c56-4e5d50e2f099-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq\" (UID: \"af0c7848-8392-473a-8c56-4e5d50e2f099\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" Jul 13 14:25:54.566170 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.566121 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nmfv\" (UniqueName: \"kubernetes.io/projected/af0c7848-8392-473a-8c56-4e5d50e2f099-kube-api-access-4nmfv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq\" (UID: \"af0c7848-8392-473a-8c56-4e5d50e2f099\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" Jul 13 14:25:54.566481 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.566458 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af0c7848-8392-473a-8c56-4e5d50e2f099-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq\" (UID: \"af0c7848-8392-473a-8c56-4e5d50e2f099\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" Jul 13 14:25:54.566525 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.566478 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af0c7848-8392-473a-8c56-4e5d50e2f099-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq\" (UID: \"af0c7848-8392-473a-8c56-4e5d50e2f099\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" Jul 13 14:25:54.577516 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.577486 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nmfv\" (UniqueName: \"kubernetes.io/projected/af0c7848-8392-473a-8c56-4e5d50e2f099-kube-api-access-4nmfv\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq\" (UID: \"af0c7848-8392-473a-8c56-4e5d50e2f099\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" Jul 13 14:25:54.630673 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.630571 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:25:54.655118 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.655092 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" Jul 13 14:25:54.783762 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.783723 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq"] Jul 13 14:25:54.786639 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:25:54.786605 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0c7848_8392_473a_8c56_4e5d50e2f099.slice/crio-7f2ab83a040d759e1db1c1832c6b6f357d084e96c2e73793701c5e493f0d00e0 WatchSource:0}: Error finding container 7f2ab83a040d759e1db1c1832c6b6f357d084e96c2e73793701c5e493f0d00e0: Status 404 returned error can't find the container with id 7f2ab83a040d759e1db1c1832c6b6f357d084e96c2e73793701c5e493f0d00e0 Jul 13 14:25:54.943801 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.943719 2566 generic.go:356] "Generic (PLEG): container finished" podID="af0c7848-8392-473a-8c56-4e5d50e2f099" containerID="6a3af5ad0f34af50010f8bc97931592163c56de3559ad9ee95f54ee2ec3952c1" exitCode=0 Jul 13 14:25:54.943945 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.943810 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" event={"ID":"af0c7848-8392-473a-8c56-4e5d50e2f099","Type":"ContainerDied","Data":"6a3af5ad0f34af50010f8bc97931592163c56de3559ad9ee95f54ee2ec3952c1"} Jul 13 14:25:54.943945 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:54.943870 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" event={"ID":"af0c7848-8392-473a-8c56-4e5d50e2f099","Type":"ContainerStarted","Data":"7f2ab83a040d759e1db1c1832c6b6f357d084e96c2e73793701c5e493f0d00e0"} Jul 13 14:25:55.949550 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:55.949519 2566 generic.go:356] "Generic (PLEG): container finished" podID="af0c7848-8392-473a-8c56-4e5d50e2f099" containerID="79abd99e57502f220f8cb6c9a9609d1c7fad1ba1485f350be567841c9bb3273e" exitCode=0 Jul 13 14:25:55.949889 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:55.949562 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" event={"ID":"af0c7848-8392-473a-8c56-4e5d50e2f099","Type":"ContainerDied","Data":"79abd99e57502f220f8cb6c9a9609d1c7fad1ba1485f350be567841c9bb3273e"} Jul 13 14:25:56.955302 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:56.955268 2566 generic.go:356] "Generic (PLEG): container finished" podID="af0c7848-8392-473a-8c56-4e5d50e2f099" containerID="8f5ce5633cf870280cb1495e6d07c27b8ac0a882b3ee9afa2c54c23032178b6c" exitCode=0 Jul 13 14:25:56.955668 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:56.955312 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" event={"ID":"af0c7848-8392-473a-8c56-4e5d50e2f099","Type":"ContainerDied","Data":"8f5ce5633cf870280cb1495e6d07c27b8ac0a882b3ee9afa2c54c23032178b6c"} Jul 13 14:25:58.077937 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:58.077912 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" Jul 13 14:25:58.192660 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:58.192628 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af0c7848-8392-473a-8c56-4e5d50e2f099-util\") pod \"af0c7848-8392-473a-8c56-4e5d50e2f099\" (UID: \"af0c7848-8392-473a-8c56-4e5d50e2f099\") " Jul 13 14:25:58.192821 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:58.192705 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nmfv\" (UniqueName: \"kubernetes.io/projected/af0c7848-8392-473a-8c56-4e5d50e2f099-kube-api-access-4nmfv\") pod \"af0c7848-8392-473a-8c56-4e5d50e2f099\" (UID: \"af0c7848-8392-473a-8c56-4e5d50e2f099\") " Jul 13 14:25:58.192821 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:58.192727 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af0c7848-8392-473a-8c56-4e5d50e2f099-bundle\") pod \"af0c7848-8392-473a-8c56-4e5d50e2f099\" (UID: \"af0c7848-8392-473a-8c56-4e5d50e2f099\") " Jul 13 14:25:58.193569 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:58.193541 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0c7848-8392-473a-8c56-4e5d50e2f099-bundle" (OuterVolumeSpecName: "bundle") pod "af0c7848-8392-473a-8c56-4e5d50e2f099" (UID: "af0c7848-8392-473a-8c56-4e5d50e2f099"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:25:58.194814 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:58.194779 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0c7848-8392-473a-8c56-4e5d50e2f099-kube-api-access-4nmfv" (OuterVolumeSpecName: "kube-api-access-4nmfv") pod "af0c7848-8392-473a-8c56-4e5d50e2f099" (UID: "af0c7848-8392-473a-8c56-4e5d50e2f099"). InnerVolumeSpecName "kube-api-access-4nmfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:25:58.198992 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:58.198961 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0c7848-8392-473a-8c56-4e5d50e2f099-util" (OuterVolumeSpecName: "util") pod "af0c7848-8392-473a-8c56-4e5d50e2f099" (UID: "af0c7848-8392-473a-8c56-4e5d50e2f099"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:25:58.293735 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:58.293675 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nmfv\" (UniqueName: \"kubernetes.io/projected/af0c7848-8392-473a-8c56-4e5d50e2f099-kube-api-access-4nmfv\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:25:58.293735 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:58.293696 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/af0c7848-8392-473a-8c56-4e5d50e2f099-bundle\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:25:58.293735 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:58.293705 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/af0c7848-8392-473a-8c56-4e5d50e2f099-util\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:25:58.746499 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:58.746471 2566 kubelet.go:1648] "Image garbage collection succeeded" Jul 13 14:25:58.963647 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:58.963615 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" event={"ID":"af0c7848-8392-473a-8c56-4e5d50e2f099","Type":"ContainerDied","Data":"7f2ab83a040d759e1db1c1832c6b6f357d084e96c2e73793701c5e493f0d00e0"} Jul 13 14:25:58.963647 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:58.963642 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af74qcq" Jul 13 14:25:58.963872 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:25:58.963666 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f2ab83a040d759e1db1c1832c6b6f357d084e96c2e73793701c5e493f0d00e0" Jul 13 14:26:08.806681 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:08.806591 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:10.121313 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:10.121278 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:10.464293 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:10.464164 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:10.558893 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:10.558862 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:11.213082 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.213032 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:11.631311 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.631282 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:11.781325 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.781284 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k"] Jul 13 14:26:11.781545 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.781534 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af0c7848-8392-473a-8c56-4e5d50e2f099" containerName="util" Jul 13 14:26:11.781595 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.781547 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="af0c7848-8392-473a-8c56-4e5d50e2f099" containerName="util" Jul 13 14:26:11.781595 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.781561 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af0c7848-8392-473a-8c56-4e5d50e2f099" containerName="pull" Jul 13 14:26:11.781595 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.781567 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="af0c7848-8392-473a-8c56-4e5d50e2f099" containerName="pull" Jul 13 14:26:11.781595 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.781577 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af0c7848-8392-473a-8c56-4e5d50e2f099" containerName="extract" Jul 13 14:26:11.781595 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.781583 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="af0c7848-8392-473a-8c56-4e5d50e2f099" containerName="extract" Jul 13 14:26:11.781738 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.781617 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="af0c7848-8392-473a-8c56-4e5d50e2f099" containerName="extract" Jul 13 14:26:11.783509 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.783490 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:11.786025 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.786000 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f2db\" (UniqueName: \"kubernetes.io/projected/9651f48e-4df1-4b58-b29f-a08590d20e04-kube-api-access-6f2db\") pod \"opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k\" (UID: \"9651f48e-4df1-4b58-b29f-a08590d20e04\") " pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:11.786161 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.786039 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9651f48e-4df1-4b58-b29f-a08590d20e04-webhook-cert\") pod \"opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k\" (UID: \"9651f48e-4df1-4b58-b29f-a08590d20e04\") " pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:11.786161 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.786067 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9651f48e-4df1-4b58-b29f-a08590d20e04-tmp\") pod \"opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k\" (UID: \"9651f48e-4df1-4b58-b29f-a08590d20e04\") " pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:11.786244 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.786174 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9651f48e-4df1-4b58-b29f-a08590d20e04-manifests\") pod \"opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k\" (UID: \"9651f48e-4df1-4b58-b29f-a08590d20e04\") " pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:11.786244 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.786220 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9651f48e-4df1-4b58-b29f-a08590d20e04-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k\" (UID: \"9651f48e-4df1-4b58-b29f-a08590d20e04\") " pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:11.789995 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.789973 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-769rd\"" Jul 13 14:26:11.790103 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.789973 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 13 14:26:11.790348 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.790323 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 13 14:26:11.790440 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.790381 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 13 14:26:11.790440 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.790408 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 13 14:26:11.810745 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.810721 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k"] Jul 13 14:26:11.821708 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.821682 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:11.887276 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.887204 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9651f48e-4df1-4b58-b29f-a08590d20e04-manifests\") pod \"opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k\" (UID: \"9651f48e-4df1-4b58-b29f-a08590d20e04\") " pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:11.887276 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.887236 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9651f48e-4df1-4b58-b29f-a08590d20e04-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k\" (UID: \"9651f48e-4df1-4b58-b29f-a08590d20e04\") " pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:11.887276 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.887272 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f2db\" (UniqueName: \"kubernetes.io/projected/9651f48e-4df1-4b58-b29f-a08590d20e04-kube-api-access-6f2db\") pod \"opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k\" (UID: \"9651f48e-4df1-4b58-b29f-a08590d20e04\") " pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:11.887531 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.887290 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9651f48e-4df1-4b58-b29f-a08590d20e04-webhook-cert\") pod \"opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k\" (UID: \"9651f48e-4df1-4b58-b29f-a08590d20e04\") " pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:11.887531 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.887305 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9651f48e-4df1-4b58-b29f-a08590d20e04-tmp\") pod \"opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k\" (UID: \"9651f48e-4df1-4b58-b29f-a08590d20e04\") " pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:11.887660 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.887637 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9651f48e-4df1-4b58-b29f-a08590d20e04-manifests\") pod \"opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k\" (UID: \"9651f48e-4df1-4b58-b29f-a08590d20e04\") " pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:11.889472 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.889450 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9651f48e-4df1-4b58-b29f-a08590d20e04-tmp\") pod \"opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k\" (UID: \"9651f48e-4df1-4b58-b29f-a08590d20e04\") " pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:11.889876 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.889856 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9651f48e-4df1-4b58-b29f-a08590d20e04-webhook-cert\") pod \"opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k\" (UID: \"9651f48e-4df1-4b58-b29f-a08590d20e04\") " pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:11.889925 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.889856 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9651f48e-4df1-4b58-b29f-a08590d20e04-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k\" (UID: \"9651f48e-4df1-4b58-b29f-a08590d20e04\") " pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:11.905485 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:11.905455 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f2db\" (UniqueName: \"kubernetes.io/projected/9651f48e-4df1-4b58-b29f-a08590d20e04-kube-api-access-6f2db\") pod \"opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k\" (UID: \"9651f48e-4df1-4b58-b29f-a08590d20e04\") " pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:12.050250 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:12.050201 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:12.093345 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:12.093309 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:12.163689 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:12.163664 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:12.234306 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:12.234280 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k"] Jul 13 14:26:12.237680 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:26:12.237653 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9651f48e_4df1_4b58_b29f_a08590d20e04.slice/crio-c246a37188ac720b06928d2b88c9156ffa1b038a3dbde25757ea83458652822f WatchSource:0}: Error finding container c246a37188ac720b06928d2b88c9156ffa1b038a3dbde25757ea83458652822f: Status 404 returned error can't find the container with id c246a37188ac720b06928d2b88c9156ffa1b038a3dbde25757ea83458652822f Jul 13 14:26:12.238606 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:12.238583 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:12.239355 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:12.239336 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 14:26:12.317620 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:12.317593 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:12.423830 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:12.423735 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:12.541103 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:12.541074 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:12.648980 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:12.648947 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:13.016852 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:13.016815 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" event={"ID":"9651f48e-4df1-4b58-b29f-a08590d20e04","Type":"ContainerStarted","Data":"c246a37188ac720b06928d2b88c9156ffa1b038a3dbde25757ea83458652822f"} Jul 13 14:26:16.030605 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:16.030564 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" event={"ID":"9651f48e-4df1-4b58-b29f-a08590d20e04","Type":"ContainerStarted","Data":"349f9f1fd94d9d3d8779e462771084c86573bf4155ff2e4ff32771a0ba584768"} Jul 13 14:26:17.035274 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:17.035235 2566 generic.go:356] "Generic (PLEG): container finished" podID="9651f48e-4df1-4b58-b29f-a08590d20e04" containerID="349f9f1fd94d9d3d8779e462771084c86573bf4155ff2e4ff32771a0ba584768" exitCode=0 Jul 13 14:26:17.035622 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:17.035284 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" event={"ID":"9651f48e-4df1-4b58-b29f-a08590d20e04","Type":"ContainerDied","Data":"349f9f1fd94d9d3d8779e462771084c86573bf4155ff2e4ff32771a0ba584768"} Jul 13 14:26:18.041095 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:18.041058 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" event={"ID":"9651f48e-4df1-4b58-b29f-a08590d20e04","Type":"ContainerStarted","Data":"8dc3e666d4db00b37ae2d9548b7d60cd83ad8721f7ce29ffd06b33eaa487e802"} Jul 13 14:26:18.041518 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:18.041167 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:18.093348 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:18.093294 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" podStartSLOduration=2.751651283 podStartE2EDuration="7.093278346s" totalImagesPullingTime="3.332215337s" totalInitContainerRuntime="1.009411726s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:26:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:26:18.089976306 +0000 UTC m=+319.774396625" watchObservedRunningTime="2026-07-13 14:26:18.093278346 +0000 UTC m=+319.777698653" Jul 13 14:26:23.257223 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:23.257189 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:23.359534 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:23.359504 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:23.668333 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:23.668302 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:23.768215 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:23.768187 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:29.046957 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:29.046928 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k" Jul 13 14:26:33.240286 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:33.240252 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:33.416068 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:33.416039 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:33.988072 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:33.988036 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:34.024450 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.024416 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz"] Jul 13 14:26:34.033464 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.033431 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz" Jul 13 14:26:34.038772 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.038750 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 13 14:26:34.038909 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.038750 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-lnhsf\"" Jul 13 14:26:34.038909 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.038808 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 13 14:26:34.039461 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.039445 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 13 14:26:34.043026 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.043010 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 13 14:26:34.069399 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.069371 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz"] Jul 13 14:26:34.125340 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.125308 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv6lz\" (UniqueName: \"kubernetes.io/projected/2e8f8592-fe37-438a-8d5a-c9522c404a05-kube-api-access-rv6lz\") pod \"kube-auth-proxy-7fcc85dc7-crddz\" (UID: \"2e8f8592-fe37-438a-8d5a-c9522c404a05\") " pod="openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz" Jul 13 14:26:34.125499 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.125348 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8f8592-fe37-438a-8d5a-c9522c404a05-tls-certs\") pod \"kube-auth-proxy-7fcc85dc7-crddz\" (UID: \"2e8f8592-fe37-438a-8d5a-c9522c404a05\") " pod="openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz" Jul 13 14:26:34.125499 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.125369 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e8f8592-fe37-438a-8d5a-c9522c404a05-tmp\") pod \"kube-auth-proxy-7fcc85dc7-crddz\" (UID: \"2e8f8592-fe37-438a-8d5a-c9522c404a05\") " pod="openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz" Jul 13 14:26:34.212480 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.212448 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:34.226555 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.226530 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8f8592-fe37-438a-8d5a-c9522c404a05-tls-certs\") pod \"kube-auth-proxy-7fcc85dc7-crddz\" (UID: \"2e8f8592-fe37-438a-8d5a-c9522c404a05\") " pod="openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz" Jul 13 14:26:34.226665 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.226557 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e8f8592-fe37-438a-8d5a-c9522c404a05-tmp\") pod \"kube-auth-proxy-7fcc85dc7-crddz\" (UID: \"2e8f8592-fe37-438a-8d5a-c9522c404a05\") " pod="openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz" Jul 13 14:26:34.226665 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.226611 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv6lz\" (UniqueName: \"kubernetes.io/projected/2e8f8592-fe37-438a-8d5a-c9522c404a05-kube-api-access-rv6lz\") pod \"kube-auth-proxy-7fcc85dc7-crddz\" (UID: \"2e8f8592-fe37-438a-8d5a-c9522c404a05\") " pod="openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz" Jul 13 14:26:34.228878 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.228859 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e8f8592-fe37-438a-8d5a-c9522c404a05-tmp\") pod \"kube-auth-proxy-7fcc85dc7-crddz\" (UID: \"2e8f8592-fe37-438a-8d5a-c9522c404a05\") " pod="openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz" Jul 13 14:26:34.229165 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.229146 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e8f8592-fe37-438a-8d5a-c9522c404a05-tls-certs\") pod \"kube-auth-proxy-7fcc85dc7-crddz\" (UID: \"2e8f8592-fe37-438a-8d5a-c9522c404a05\") " pod="openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz" Jul 13 14:26:34.243097 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.243037 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r5gm9"] Jul 13 14:26:34.244090 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.244071 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv6lz\" (UniqueName: \"kubernetes.io/projected/2e8f8592-fe37-438a-8d5a-c9522c404a05-kube-api-access-rv6lz\") pod \"kube-auth-proxy-7fcc85dc7-crddz\" (UID: \"2e8f8592-fe37-438a-8d5a-c9522c404a05\") " pod="openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz" Jul 13 14:26:34.245215 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.245199 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r5gm9" Jul 13 14:26:34.248683 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.248665 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5mj5s\"" Jul 13 14:26:34.249182 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.249161 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 13 14:26:34.295800 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.295757 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r5gm9"] Jul 13 14:26:34.326906 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.326869 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7cvh\" (UniqueName: \"kubernetes.io/projected/b2395092-7b71-4dcf-9478-cc9676ac3c40-kube-api-access-r7cvh\") pod \"odh-model-controller-7998c67dc4-r5gm9\" (UID: \"b2395092-7b71-4dcf-9478-cc9676ac3c40\") " pod="opendatahub/odh-model-controller-7998c67dc4-r5gm9" Jul 13 14:26:34.327037 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.326919 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2395092-7b71-4dcf-9478-cc9676ac3c40-cert\") pod \"odh-model-controller-7998c67dc4-r5gm9\" (UID: \"b2395092-7b71-4dcf-9478-cc9676ac3c40\") " pod="opendatahub/odh-model-controller-7998c67dc4-r5gm9" Jul 13 14:26:34.342899 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.342877 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz" Jul 13 14:26:34.405606 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.405570 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:34.428326 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.428301 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2395092-7b71-4dcf-9478-cc9676ac3c40-cert\") pod \"odh-model-controller-7998c67dc4-r5gm9\" (UID: \"b2395092-7b71-4dcf-9478-cc9676ac3c40\") " pod="opendatahub/odh-model-controller-7998c67dc4-r5gm9" Jul 13 14:26:34.428469 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.428377 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7cvh\" (UniqueName: \"kubernetes.io/projected/b2395092-7b71-4dcf-9478-cc9676ac3c40-kube-api-access-r7cvh\") pod \"odh-model-controller-7998c67dc4-r5gm9\" (UID: \"b2395092-7b71-4dcf-9478-cc9676ac3c40\") " pod="opendatahub/odh-model-controller-7998c67dc4-r5gm9" Jul 13 14:26:34.428664 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:26:34.428638 2566 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 13 14:26:34.428800 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:26:34.428716 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2395092-7b71-4dcf-9478-cc9676ac3c40-cert podName:b2395092-7b71-4dcf-9478-cc9676ac3c40 nodeName:}" failed. No retries permitted until 2026-07-13 14:26:34.928691158 +0000 UTC m=+336.613111447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b2395092-7b71-4dcf-9478-cc9676ac3c40-cert") pod "odh-model-controller-7998c67dc4-r5gm9" (UID: "b2395092-7b71-4dcf-9478-cc9676ac3c40") : secret "odh-model-controller-webhook-cert" not found Jul 13 14:26:34.466861 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.466834 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7cvh\" (UniqueName: \"kubernetes.io/projected/b2395092-7b71-4dcf-9478-cc9676ac3c40-kube-api-access-r7cvh\") pod \"odh-model-controller-7998c67dc4-r5gm9\" (UID: \"b2395092-7b71-4dcf-9478-cc9676ac3c40\") " pod="opendatahub/odh-model-controller-7998c67dc4-r5gm9" Jul 13 14:26:34.505131 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:26:34.505090 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8f8592_fe37_438a_8d5a_c9522c404a05.slice/crio-f2c5827c2ac1e408c66b9e2844fd7608ee15903f51a71b00fe5bbd44f5a6ecef WatchSource:0}: Error finding container f2c5827c2ac1e408c66b9e2844fd7608ee15903f51a71b00fe5bbd44f5a6ecef: Status 404 returned error can't find the container with id f2c5827c2ac1e408c66b9e2844fd7608ee15903f51a71b00fe5bbd44f5a6ecef Jul 13 14:26:34.507023 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.507001 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz"] Jul 13 14:26:34.519580 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.519560 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:34.818472 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.818445 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:34.931765 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:34.931724 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2395092-7b71-4dcf-9478-cc9676ac3c40-cert\") pod \"odh-model-controller-7998c67dc4-r5gm9\" (UID: \"b2395092-7b71-4dcf-9478-cc9676ac3c40\") " pod="opendatahub/odh-model-controller-7998c67dc4-r5gm9" Jul 13 14:26:34.931934 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:26:34.931876 2566 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 13 14:26:34.931934 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:26:34.931930 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2395092-7b71-4dcf-9478-cc9676ac3c40-cert podName:b2395092-7b71-4dcf-9478-cc9676ac3c40 nodeName:}" failed. No retries permitted until 2026-07-13 14:26:35.931916398 +0000 UTC m=+337.616336686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b2395092-7b71-4dcf-9478-cc9676ac3c40-cert") pod "odh-model-controller-7998c67dc4-r5gm9" (UID: "b2395092-7b71-4dcf-9478-cc9676ac3c40") : secret "odh-model-controller-webhook-cert" not found Jul 13 14:26:35.102623 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:35.102548 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz" event={"ID":"2e8f8592-fe37-438a-8d5a-c9522c404a05","Type":"ContainerStarted","Data":"f2c5827c2ac1e408c66b9e2844fd7608ee15903f51a71b00fe5bbd44f5a6ecef"} Jul 13 14:26:35.454055 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:35.453979 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:35.808910 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:35.808831 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:35.887691 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:35.887660 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9jkwn"] Jul 13 14:26:35.889847 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:35.889831 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9jkwn" Jul 13 14:26:35.893462 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:35.893444 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 13 14:26:35.905915 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:35.905883 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9jkwn"] Jul 13 14:26:35.939566 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:35.939536 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2395092-7b71-4dcf-9478-cc9676ac3c40-cert\") pod \"odh-model-controller-7998c67dc4-r5gm9\" (UID: \"b2395092-7b71-4dcf-9478-cc9676ac3c40\") " pod="opendatahub/odh-model-controller-7998c67dc4-r5gm9" Jul 13 14:26:35.939720 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:35.939624 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdwcn\" (UniqueName: \"kubernetes.io/projected/17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb-kube-api-access-tdwcn\") pod \"model-serving-api-54d89bd5fc-9jkwn\" (UID: \"17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb\") " pod="opendatahub/model-serving-api-54d89bd5fc-9jkwn" Jul 13 14:26:35.939720 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:35.939661 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb-tls-certs\") pod \"model-serving-api-54d89bd5fc-9jkwn\" (UID: \"17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb\") " pod="opendatahub/model-serving-api-54d89bd5fc-9jkwn" Jul 13 14:26:35.942197 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:35.942177 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2395092-7b71-4dcf-9478-cc9676ac3c40-cert\") pod \"odh-model-controller-7998c67dc4-r5gm9\" (UID: \"b2395092-7b71-4dcf-9478-cc9676ac3c40\") " pod="opendatahub/odh-model-controller-7998c67dc4-r5gm9" Jul 13 14:26:35.949267 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:35.949224 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:35.990936 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:35.990908 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5"] Jul 13 14:26:35.993281 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:35.993259 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.002093 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.002073 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 13 14:26:36.002518 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.002495 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 13 14:26:36.002771 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.002753 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 13 14:26:36.002898 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.002800 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-r8lrb\"" Jul 13 14:26:36.003056 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.003040 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 13 14:26:36.010192 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.010174 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5"] Jul 13 14:26:36.040243 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.040218 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.040348 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.040259 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-token\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.040348 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.040295 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-cacerts\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.040428 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.040343 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7k9\" (UniqueName: \"kubernetes.io/projected/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-kube-api-access-8r7k9\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.040428 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.040379 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-local-certs\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.040428 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.040412 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwcn\" (UniqueName: \"kubernetes.io/projected/17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb-kube-api-access-tdwcn\") pod \"model-serving-api-54d89bd5fc-9jkwn\" (UID: \"17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb\") " pod="opendatahub/model-serving-api-54d89bd5fc-9jkwn" Jul 13 14:26:36.040544 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.040442 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb-tls-certs\") pod \"model-serving-api-54d89bd5fc-9jkwn\" (UID: \"17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb\") " pod="opendatahub/model-serving-api-54d89bd5fc-9jkwn" Jul 13 14:26:36.040544 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.040478 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.040694 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:26:36.040678 2566 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 13 14:26:36.040728 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.040711 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.040772 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:26:36.040753 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb-tls-certs podName:17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb nodeName:}" failed. No retries permitted until 2026-07-13 14:26:36.540733067 +0000 UTC m=+338.225153354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb-tls-certs") pod "model-serving-api-54d89bd5fc-9jkwn" (UID: "17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb") : secret "model-serving-api-tls" not found Jul 13 14:26:36.051304 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.051282 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwcn\" (UniqueName: \"kubernetes.io/projected/17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb-kube-api-access-tdwcn\") pod \"model-serving-api-54d89bd5fc-9jkwn\" (UID: \"17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb\") " pod="opendatahub/model-serving-api-54d89bd5fc-9jkwn" Jul 13 14:26:36.058073 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.058056 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r5gm9" Jul 13 14:26:36.141698 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.141672 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.141837 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.141704 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.141837 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.141733 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.141837 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.141762 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-token\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.141837 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.141818 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-cacerts\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.142051 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.141841 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7k9\" (UniqueName: \"kubernetes.io/projected/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-kube-api-access-8r7k9\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.142051 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.141872 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-local-certs\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.142491 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.142412 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.144235 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.144205 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-local-certs\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.144378 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.144354 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.144490 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.144468 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.144551 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.144537 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-cacerts\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.152077 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.152051 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-token\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.155273 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.155238 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7k9\" (UniqueName: \"kubernetes.io/projected/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-kube-api-access-8r7k9\") pod \"istiod-openshift-gateway-7574459656-fjqf5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.205109 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.205081 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r5gm9"] Jul 13 14:26:36.208122 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:26:36.208091 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2395092_7b71_4dcf_9478_cc9676ac3c40.slice/crio-3f37ca8fa992bf6c920abdba1a557176efddec2bf14fa6f39dc11256a8c5a103 WatchSource:0}: Error finding container 3f37ca8fa992bf6c920abdba1a557176efddec2bf14fa6f39dc11256a8c5a103: Status 404 returned error can't find the container with id 3f37ca8fa992bf6c920abdba1a557176efddec2bf14fa6f39dc11256a8c5a103 Jul 13 14:26:36.304742 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.304703 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:36.450346 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.450310 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5"] Jul 13 14:26:36.457164 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:26:36.457130 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1b7b79_cd3f_4160_8ae3_b0da21d13bd5.slice/crio-babc96fb768df188741e4efb6bfe3ead163f739ee5ac5a6c6954d869ce26f529 WatchSource:0}: Error finding container babc96fb768df188741e4efb6bfe3ead163f739ee5ac5a6c6954d869ce26f529: Status 404 returned error can't find the container with id babc96fb768df188741e4efb6bfe3ead163f739ee5ac5a6c6954d869ce26f529 Jul 13 14:26:36.545697 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.545666 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb-tls-certs\") pod \"model-serving-api-54d89bd5fc-9jkwn\" (UID: \"17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb\") " pod="opendatahub/model-serving-api-54d89bd5fc-9jkwn" Jul 13 14:26:36.548012 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.547989 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb-tls-certs\") pod \"model-serving-api-54d89bd5fc-9jkwn\" (UID: \"17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb\") " pod="opendatahub/model-serving-api-54d89bd5fc-9jkwn" Jul 13 14:26:36.626779 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.626700 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:36.746141 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.746110 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:36.801317 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:36.801286 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9jkwn" Jul 13 14:26:37.112699 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:37.112665 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" event={"ID":"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5","Type":"ContainerStarted","Data":"babc96fb768df188741e4efb6bfe3ead163f739ee5ac5a6c6954d869ce26f529"} Jul 13 14:26:37.114074 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:37.114041 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r5gm9" event={"ID":"b2395092-7b71-4dcf-9478-cc9676ac3c40","Type":"ContainerStarted","Data":"3f37ca8fa992bf6c920abdba1a557176efddec2bf14fa6f39dc11256a8c5a103"} Jul 13 14:26:37.325226 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:37.325108 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9jkwn"] Jul 13 14:26:37.327621 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:26:37.327595 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e3ff33_cce6_4a3a_896b_f2f1ba02eaeb.slice/crio-cbc4ce6582826569abd42a7c4eccd3e390792cb276f462ecd0ba1c3883fef20e WatchSource:0}: Error finding container cbc4ce6582826569abd42a7c4eccd3e390792cb276f462ecd0ba1c3883fef20e: Status 404 returned error can't find the container with id cbc4ce6582826569abd42a7c4eccd3e390792cb276f462ecd0ba1c3883fef20e Jul 13 14:26:37.934011 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:37.933981 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:38.120400 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:38.120315 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9jkwn" event={"ID":"17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb","Type":"ContainerStarted","Data":"cbc4ce6582826569abd42a7c4eccd3e390792cb276f462ecd0ba1c3883fef20e"} Jul 13 14:26:38.122468 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:38.122042 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz" event={"ID":"2e8f8592-fe37-438a-8d5a-c9522c404a05","Type":"ContainerStarted","Data":"670d7e26a3b24bf3bfd97752af919d7ff245afcb221020f066fce7fd44a9a6c3"} Jul 13 14:26:38.157066 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:38.157007 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7fcc85dc7-crddz" podStartSLOduration=2.448532389 podStartE2EDuration="5.156982981s" totalImagesPullingTime="2.708450592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:26:38.156350112 +0000 UTC m=+339.840770418" watchObservedRunningTime="2026-07-13 14:26:38.156982981 +0000 UTC m=+339.841403288" Jul 13 14:26:40.138640 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:40.138576 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9jkwn" event={"ID":"17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb","Type":"ContainerStarted","Data":"e10db7edde81fae938ecfea79aeb6c0aa3ef02f3dd8e2cf29b20ae0c2791ed09"} Jul 13 14:26:40.139011 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:40.138671 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-9jkwn" Jul 13 14:26:40.162049 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:40.162003 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-9jkwn" podStartSLOduration=3.2772397460000002 podStartE2EDuration="5.161989104s" totalImagesPullingTime="1.884749358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:26:40.159986577 +0000 UTC m=+341.844406883" watchObservedRunningTime="2026-07-13 14:26:40.161989104 +0000 UTC m=+341.846409442" Jul 13 14:26:41.146185 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:41.146146 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r5gm9" event={"ID":"b2395092-7b71-4dcf-9478-cc9676ac3c40","Type":"ContainerStarted","Data":"23a340d464426735f5822ba6b7176e4619967117b1b602b38b9d43e51ba00c56"} Jul 13 14:26:41.146653 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:41.146527 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-r5gm9" Jul 13 14:26:42.582687 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:42.582656 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:26:42.582959 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:42.582720 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:26:43.156487 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:43.156452 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" event={"ID":"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5","Type":"ContainerStarted","Data":"66dd8b81df837fb53beb82a4aaa71fa21b4e41fc141e41234aacdac2a54e5021"} Jul 13 14:26:43.295344 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:43.295292 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-r5gm9" podStartSLOduration=4.7904576500000005 podStartE2EDuration="9.295271902s" totalImagesPullingTime="4.504814252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:26:41.186274585 +0000 UTC m=+342.870694891" watchObservedRunningTime="2026-07-13 14:26:43.295271902 +0000 UTC m=+344.979692211" Jul 13 14:26:43.295523 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:43.295450 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" podStartSLOduration=2.172154898 podStartE2EDuration="8.29543931s" totalImagesPullingTime="6.123284412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:26:43.294081329 +0000 UTC m=+344.978501664" watchObservedRunningTime="2026-07-13 14:26:43.29543931 +0000 UTC m=+344.979859616" Jul 13 14:26:43.893818 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:43.893727 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:43.996577 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:43.996537 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:44.161429 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.161356 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:44.162845 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.162817 2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-fjqf5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 14:26:44.162963 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.162873 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" podUID="7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 14:26:44.601284 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.601252 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk"] Jul 13 14:26:44.603376 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.603359 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.606188 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.606138 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:44.611672 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.611655 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-r22hj\"" Jul 13 14:26:44.627493 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.627468 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk"] Jul 13 14:26:44.715706 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.715671 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0410a37e-f431-4f7c-98c6-4b33ccc893fb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.715839 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.715709 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9gh\" (UniqueName: \"kubernetes.io/projected/0410a37e-f431-4f7c-98c6-4b33ccc893fb-kube-api-access-9b9gh\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.715839 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.715730 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410a37e-f431-4f7c-98c6-4b33ccc893fb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.715839 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.715752 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0410a37e-f431-4f7c-98c6-4b33ccc893fb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.715839 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.715825 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0410a37e-f431-4f7c-98c6-4b33ccc893fb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.715979 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.715865 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0410a37e-f431-4f7c-98c6-4b33ccc893fb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.715979 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.715882 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0410a37e-f431-4f7c-98c6-4b33ccc893fb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.715979 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.715966 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410a37e-f431-4f7c-98c6-4b33ccc893fb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.716070 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.715995 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0410a37e-f431-4f7c-98c6-4b33ccc893fb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.816584 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.816536 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410a37e-f431-4f7c-98c6-4b33ccc893fb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.816584 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.816582 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0410a37e-f431-4f7c-98c6-4b33ccc893fb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.816839 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.816608 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0410a37e-f431-4f7c-98c6-4b33ccc893fb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.816839 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.816639 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9gh\" (UniqueName: \"kubernetes.io/projected/0410a37e-f431-4f7c-98c6-4b33ccc893fb-kube-api-access-9b9gh\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.816839 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.816663 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410a37e-f431-4f7c-98c6-4b33ccc893fb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.816839 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.816686 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0410a37e-f431-4f7c-98c6-4b33ccc893fb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.817001 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.816861 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0410a37e-f431-4f7c-98c6-4b33ccc893fb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.817001 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.816916 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0410a37e-f431-4f7c-98c6-4b33ccc893fb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.817001 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.816952 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0410a37e-f431-4f7c-98c6-4b33ccc893fb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.817461 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.817430 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410a37e-f431-4f7c-98c6-4b33ccc893fb-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.817461 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.817444 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0410a37e-f431-4f7c-98c6-4b33ccc893fb-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.817649 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.817509 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0410a37e-f431-4f7c-98c6-4b33ccc893fb-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.817649 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.817505 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0410a37e-f431-4f7c-98c6-4b33ccc893fb-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.818113 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.818086 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0410a37e-f431-4f7c-98c6-4b33ccc893fb-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.819966 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.819941 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0410a37e-f431-4f7c-98c6-4b33ccc893fb-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.823134 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.823108 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0410a37e-f431-4f7c-98c6-4b33ccc893fb-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.839749 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.839716 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0410a37e-f431-4f7c-98c6-4b33ccc893fb-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.839905 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.839870 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9gh\" (UniqueName: \"kubernetes.io/projected/0410a37e-f431-4f7c-98c6-4b33ccc893fb-kube-api-access-9b9gh\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4mhpqk\" (UID: \"0410a37e-f431-4f7c-98c6-4b33ccc893fb\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:44.913640 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:44.913566 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:45.061524 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:45.061485 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:45.078436 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:45.078413 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk"] Jul 13 14:26:45.080183 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:26:45.080158 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0410a37e_f431_4f7c_98c6_4b33ccc893fb.slice/crio-34c6bdd2c717bab130f528af53b2c41ae8118b71e1ae2b5ac3aa20f2222038cf WatchSource:0}: Error finding container 34c6bdd2c717bab130f528af53b2c41ae8118b71e1ae2b5ac3aa20f2222038cf: Status 404 returned error can't find the container with id 34c6bdd2c717bab130f528af53b2c41ae8118b71e1ae2b5ac3aa20f2222038cf Jul 13 14:26:45.165766 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:45.165700 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" event={"ID":"0410a37e-f431-4f7c-98c6-4b33ccc893fb","Type":"ContainerStarted","Data":"34c6bdd2c717bab130f528af53b2c41ae8118b71e1ae2b5ac3aa20f2222038cf"} Jul 13 14:26:45.166396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:45.166377 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:45.176407 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:45.176388 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:47.419823 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.419777 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:26:47.420066 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.419862 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:26:47.420066 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.419893 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:26:47.610502 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.610461 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5"] Jul 13 14:26:47.610732 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.610694 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" podUID="7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5" containerName="discovery" containerID="cri-o://66dd8b81df837fb53beb82a4aaa71fa21b4e41fc141e41234aacdac2a54e5021" gracePeriod=30 Jul 13 14:26:47.844455 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.844427 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:47.943634 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.943603 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-token\") pod \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " Jul 13 14:26:47.943825 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.943659 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-cacerts\") pod \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " Jul 13 14:26:47.943825 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.943692 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-csr-dns-cert\") pod \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " Jul 13 14:26:47.943825 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.943725 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-kubeconfig\") pod \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " Jul 13 14:26:47.943825 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.943752 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r7k9\" (UniqueName: \"kubernetes.io/projected/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-kube-api-access-8r7k9\") pod \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " Jul 13 14:26:47.943825 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.943801 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-local-certs\") pod \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " Jul 13 14:26:47.944072 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.943828 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-csr-ca-configmap\") pod \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\" (UID: \"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5\") " Jul 13 14:26:47.944398 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.944340 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5" (UID: "7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:26:47.946046 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.946018 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5" (UID: "7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:26:47.946340 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.946102 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-cacerts" (OuterVolumeSpecName: "cacerts") pod "7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5" (UID: "7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:26:47.946340 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.946160 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5" (UID: "7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:26:47.946340 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.946207 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5" (UID: "7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:26:47.946340 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.946286 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-kube-api-access-8r7k9" (OuterVolumeSpecName: "kube-api-access-8r7k9") pod "7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5" (UID: "7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5"). InnerVolumeSpecName "kube-api-access-8r7k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:26:47.946590 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:47.946574 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-local-certs" (OuterVolumeSpecName: "local-certs") pod "7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5" (UID: "7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:26:48.044803 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.044722 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-token\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:26:48.044803 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.044745 2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-cacerts\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:26:48.044803 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.044753 2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-csr-dns-cert\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:26:48.044803 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.044761 2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-kubeconfig\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:26:48.044803 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.044770 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r7k9\" (UniqueName: \"kubernetes.io/projected/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-kube-api-access-8r7k9\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:26:48.044803 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.044778 2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-local-certs\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:26:48.044803 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.044803 2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5-istio-csr-ca-configmap\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:26:48.183315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.183279 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" event={"ID":"0410a37e-f431-4f7c-98c6-4b33ccc893fb","Type":"ContainerStarted","Data":"0745eafde5f63801fd2b0955b0ccde25927da45c1d81c19f9f1f40b768107fc8"} Jul 13 14:26:48.184715 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.184680 2566 generic.go:356] "Generic (PLEG): container finished" podID="7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5" containerID="66dd8b81df837fb53beb82a4aaa71fa21b4e41fc141e41234aacdac2a54e5021" exitCode=0 Jul 13 14:26:48.184857 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.184718 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" Jul 13 14:26:48.184857 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.184737 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" event={"ID":"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5","Type":"ContainerDied","Data":"66dd8b81df837fb53beb82a4aaa71fa21b4e41fc141e41234aacdac2a54e5021"} Jul 13 14:26:48.184857 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.184777 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5" event={"ID":"7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5","Type":"ContainerDied","Data":"babc96fb768df188741e4efb6bfe3ead163f739ee5ac5a6c6954d869ce26f529"} Jul 13 14:26:48.184857 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.184818 2566 scope.go:117] "RemoveContainer" containerID="66dd8b81df837fb53beb82a4aaa71fa21b4e41fc141e41234aacdac2a54e5021" Jul 13 14:26:48.194615 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.194597 2566 scope.go:117] "RemoveContainer" containerID="66dd8b81df837fb53beb82a4aaa71fa21b4e41fc141e41234aacdac2a54e5021" Jul 13 14:26:48.194912 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:26:48.194893 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66dd8b81df837fb53beb82a4aaa71fa21b4e41fc141e41234aacdac2a54e5021\": container with ID starting with 66dd8b81df837fb53beb82a4aaa71fa21b4e41fc141e41234aacdac2a54e5021 not found: ID does not exist" containerID="66dd8b81df837fb53beb82a4aaa71fa21b4e41fc141e41234aacdac2a54e5021" Jul 13 14:26:48.194957 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.194919 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66dd8b81df837fb53beb82a4aaa71fa21b4e41fc141e41234aacdac2a54e5021"} err="failed to get container status \"66dd8b81df837fb53beb82a4aaa71fa21b4e41fc141e41234aacdac2a54e5021\": rpc error: code = NotFound desc = could not find container \"66dd8b81df837fb53beb82a4aaa71fa21b4e41fc141e41234aacdac2a54e5021\": container with ID starting with 66dd8b81df837fb53beb82a4aaa71fa21b4e41fc141e41234aacdac2a54e5021 not found: ID does not exist" Jul 13 14:26:48.229364 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.229333 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:48.258401 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.258323 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" podStartSLOduration=1.920920658 podStartE2EDuration="4.258304607s" totalImagesPullingTime="2.337383949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:26:48.228900958 +0000 UTC m=+349.913321264" watchObservedRunningTime="2026-07-13 14:26:48.258304607 +0000 UTC m=+349.942724916" Jul 13 14:26:48.259765 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.259746 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5"] Jul 13 14:26:48.267554 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.267532 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fjqf5"] Jul 13 14:26:48.346988 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.346960 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:48.398074 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.398046 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2"] Jul 13 14:26:48.398330 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.398318 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5" containerName="discovery" Jul 13 14:26:48.398387 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.398332 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5" containerName="discovery" Jul 13 14:26:48.398387 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.398385 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5" containerName="discovery" Jul 13 14:26:48.400605 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.400589 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d"] Jul 13 14:26:48.400723 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.400708 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2" Jul 13 14:26:48.402540 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.402525 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d" Jul 13 14:26:48.405662 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.405644 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zv28r\"" Jul 13 14:26:48.406005 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.405985 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-h7w8w\"" Jul 13 14:26:48.407317 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.407298 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 13 14:26:48.407425 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.407370 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 13 14:26:48.438040 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.438012 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d"] Jul 13 14:26:48.447741 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.447721 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aeaeaa41-ec74-4ff7-8d1b-3e4288401836-cert\") pod \"kserve-controller-manager-d6f4b65b8-fmhc2\" (UID: \"aeaeaa41-ec74-4ff7-8d1b-3e4288401836\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2" Jul 13 14:26:48.447854 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.447753 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7vng\" (UniqueName: \"kubernetes.io/projected/d1e6beff-eb92-4808-aed6-5099f1626ad4-kube-api-access-t7vng\") pod \"llmisvc-controller-manager-777c6bd7d5-vpm2d\" (UID: \"d1e6beff-eb92-4808-aed6-5099f1626ad4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d" Jul 13 14:26:48.447854 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.447810 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvkr8\" (UniqueName: \"kubernetes.io/projected/aeaeaa41-ec74-4ff7-8d1b-3e4288401836-kube-api-access-zvkr8\") pod \"kserve-controller-manager-d6f4b65b8-fmhc2\" (UID: \"aeaeaa41-ec74-4ff7-8d1b-3e4288401836\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2" Jul 13 14:26:48.447854 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.447838 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1e6beff-eb92-4808-aed6-5099f1626ad4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vpm2d\" (UID: \"d1e6beff-eb92-4808-aed6-5099f1626ad4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d" Jul 13 14:26:48.468271 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.468249 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2"] Jul 13 14:26:48.483766 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.483743 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:48.549104 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.549075 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvkr8\" (UniqueName: \"kubernetes.io/projected/aeaeaa41-ec74-4ff7-8d1b-3e4288401836-kube-api-access-zvkr8\") pod \"kserve-controller-manager-d6f4b65b8-fmhc2\" (UID: \"aeaeaa41-ec74-4ff7-8d1b-3e4288401836\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2" Jul 13 14:26:48.549193 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.549118 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1e6beff-eb92-4808-aed6-5099f1626ad4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vpm2d\" (UID: \"d1e6beff-eb92-4808-aed6-5099f1626ad4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d" Jul 13 14:26:48.549193 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.549157 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aeaeaa41-ec74-4ff7-8d1b-3e4288401836-cert\") pod \"kserve-controller-manager-d6f4b65b8-fmhc2\" (UID: \"aeaeaa41-ec74-4ff7-8d1b-3e4288401836\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2" Jul 13 14:26:48.549193 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.549186 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7vng\" (UniqueName: \"kubernetes.io/projected/d1e6beff-eb92-4808-aed6-5099f1626ad4-kube-api-access-t7vng\") pod \"llmisvc-controller-manager-777c6bd7d5-vpm2d\" (UID: \"d1e6beff-eb92-4808-aed6-5099f1626ad4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d" Jul 13 14:26:48.549362 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:26:48.549341 2566 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 13 14:26:48.549415 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:26:48.549404 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeaeaa41-ec74-4ff7-8d1b-3e4288401836-cert podName:aeaeaa41-ec74-4ff7-8d1b-3e4288401836 nodeName:}" failed. No retries permitted until 2026-07-13 14:26:49.049384727 +0000 UTC m=+350.733805024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/aeaeaa41-ec74-4ff7-8d1b-3e4288401836-cert") pod "kserve-controller-manager-d6f4b65b8-fmhc2" (UID: "aeaeaa41-ec74-4ff7-8d1b-3e4288401836") : secret "kserve-webhook-server-cert" not found Jul 13 14:26:48.549503 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:26:48.549487 2566 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 13 14:26:48.549555 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:26:48.549545 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1e6beff-eb92-4808-aed6-5099f1626ad4-cert podName:d1e6beff-eb92-4808-aed6-5099f1626ad4 nodeName:}" failed. No retries permitted until 2026-07-13 14:26:49.04952657 +0000 UTC m=+350.733946858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d1e6beff-eb92-4808-aed6-5099f1626ad4-cert") pod "llmisvc-controller-manager-777c6bd7d5-vpm2d" (UID: "d1e6beff-eb92-4808-aed6-5099f1626ad4") : secret "llmisvc-webhook-server-cert" not found Jul 13 14:26:48.567995 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.567969 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvkr8\" (UniqueName: \"kubernetes.io/projected/aeaeaa41-ec74-4ff7-8d1b-3e4288401836-kube-api-access-zvkr8\") pod \"kserve-controller-manager-d6f4b65b8-fmhc2\" (UID: \"aeaeaa41-ec74-4ff7-8d1b-3e4288401836\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2" Jul 13 14:26:48.568419 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.568403 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7vng\" (UniqueName: \"kubernetes.io/projected/d1e6beff-eb92-4808-aed6-5099f1626ad4-kube-api-access-t7vng\") pod \"llmisvc-controller-manager-777c6bd7d5-vpm2d\" (UID: \"d1e6beff-eb92-4808-aed6-5099f1626ad4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d" Jul 13 14:26:48.787724 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.787644 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:48.870358 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.870316 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5" path="/var/lib/kubelet/pods/7e1b7b79-cd3f-4160-8ae3-b0da21d13bd5/volumes" Jul 13 14:26:48.914127 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.914100 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:48.918514 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.918490 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:48.923931 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:48.923905 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:49.053457 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:49.053370 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1e6beff-eb92-4808-aed6-5099f1626ad4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vpm2d\" (UID: \"d1e6beff-eb92-4808-aed6-5099f1626ad4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d" Jul 13 14:26:49.053457 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:49.053414 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aeaeaa41-ec74-4ff7-8d1b-3e4288401836-cert\") pod \"kserve-controller-manager-d6f4b65b8-fmhc2\" (UID: \"aeaeaa41-ec74-4ff7-8d1b-3e4288401836\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2" Jul 13 14:26:49.055824 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:49.055777 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1e6beff-eb92-4808-aed6-5099f1626ad4-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vpm2d\" (UID: \"d1e6beff-eb92-4808-aed6-5099f1626ad4\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d" Jul 13 14:26:49.055947 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:49.055830 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aeaeaa41-ec74-4ff7-8d1b-3e4288401836-cert\") pod \"kserve-controller-manager-d6f4b65b8-fmhc2\" (UID: \"aeaeaa41-ec74-4ff7-8d1b-3e4288401836\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2" Jul 13 14:26:49.190041 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:49.190012 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:49.191002 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:49.190980 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4mhpqk" Jul 13 14:26:49.312149 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:49.312059 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2" Jul 13 14:26:49.317725 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:49.317709 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d" Jul 13 14:26:49.454841 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:49.454810 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2"] Jul 13 14:26:49.457829 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:26:49.457803 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeaeaa41_ec74_4ff7_8d1b_3e4288401836.slice/crio-e3677aac9939655392dbe199be24551e4ef5eadda42cf3750e94d1fa0fb26c2d WatchSource:0}: Error finding container e3677aac9939655392dbe199be24551e4ef5eadda42cf3750e94d1fa0fb26c2d: Status 404 returned error can't find the container with id e3677aac9939655392dbe199be24551e4ef5eadda42cf3750e94d1fa0fb26c2d Jul 13 14:26:49.477674 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:49.477640 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d"] Jul 13 14:26:49.480109 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:26:49.480079 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd1e6beff_eb92_4808_aed6_5099f1626ad4.slice/crio-22c9dc36fb15fd78344404f537024be431958653452053927a18bade83a2d0f5 WatchSource:0}: Error finding container 22c9dc36fb15fd78344404f537024be431958653452053927a18bade83a2d0f5: Status 404 returned error can't find the container with id 22c9dc36fb15fd78344404f537024be431958653452053927a18bade83a2d0f5 Jul 13 14:26:49.655012 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:49.654982 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:49.776842 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:49.776769 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:49.887303 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:49.887273 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:26:50.195518 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:50.195482 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2" event={"ID":"aeaeaa41-ec74-4ff7-8d1b-3e4288401836","Type":"ContainerStarted","Data":"e3677aac9939655392dbe199be24551e4ef5eadda42cf3750e94d1fa0fb26c2d"} Jul 13 14:26:50.196601 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:50.196579 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d" event={"ID":"d1e6beff-eb92-4808-aed6-5099f1626ad4","Type":"ContainerStarted","Data":"22c9dc36fb15fd78344404f537024be431958653452053927a18bade83a2d0f5"} Jul 13 14:26:51.151220 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:51.151194 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-9jkwn" Jul 13 14:26:53.159661 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:53.159633 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-r5gm9" Jul 13 14:26:55.221948 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:55.221910 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2" event={"ID":"aeaeaa41-ec74-4ff7-8d1b-3e4288401836","Type":"ContainerStarted","Data":"2c875af01278a344118531eae1b20609e77b9f9798cbdccf6bd2d3ab6cda39e8"} Jul 13 14:26:55.222480 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:55.222019 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2" Jul 13 14:26:55.223482 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:55.223459 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d" event={"ID":"d1e6beff-eb92-4808-aed6-5099f1626ad4","Type":"ContainerStarted","Data":"8b5828b51c12c5d1e908fc9e2fbf1b3fb7242f5868a24c839c84e2fdbf5cd960"} Jul 13 14:26:55.223573 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:55.223505 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d" Jul 13 14:26:55.257290 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:55.257247 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2" podStartSLOduration=2.5416021349999998 podStartE2EDuration="7.257234632s" totalImagesPullingTime="4.715632497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:26:55.25613498 +0000 UTC m=+356.940555286" watchObservedRunningTime="2026-07-13 14:26:55.257234632 +0000 UTC m=+356.941654966" Jul 13 14:26:55.283757 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:26:55.283714 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d" podStartSLOduration=2.592772423 podStartE2EDuration="7.283699985s" totalImagesPullingTime="4.690927562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:26:55.281401795 +0000 UTC m=+356.965822100" watchObservedRunningTime="2026-07-13 14:26:55.283699985 +0000 UTC m=+356.968120291" Jul 13 14:27:16.724401 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:16.724351 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:27:26.230754 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:26.230727 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vpm2d" Jul 13 14:27:26.233684 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:26.233664 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-fmhc2" Jul 13 14:27:34.907835 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:34.907797 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:27:35.003231 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.003203 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:27:35.038645 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.038618 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xtkfq"] Jul 13 14:27:35.043629 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.043608 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xtkfq" Jul 13 14:27:35.046344 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.046321 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-tl9xf\"" Jul 13 14:27:35.046816 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.046777 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 13 14:27:35.047227 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.047209 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 13 14:27:35.055989 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.055962 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xtkfq"] Jul 13 14:27:35.098950 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.098913 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:27:35.119207 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.119182 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs5cd\" (UniqueName: \"kubernetes.io/projected/003164c4-6d1b-434e-880f-32860e457de8-kube-api-access-cs5cd\") pod \"kuadrant-operator-catalog-xtkfq\" (UID: \"003164c4-6d1b-434e-880f-32860e457de8\") " pod="kuadrant-system/kuadrant-operator-catalog-xtkfq" Jul 13 14:27:35.219842 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.219752 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs5cd\" (UniqueName: \"kubernetes.io/projected/003164c4-6d1b-434e-880f-32860e457de8-kube-api-access-cs5cd\") pod \"kuadrant-operator-catalog-xtkfq\" (UID: \"003164c4-6d1b-434e-880f-32860e457de8\") " pod="kuadrant-system/kuadrant-operator-catalog-xtkfq" Jul 13 14:27:35.231560 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.231533 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs5cd\" (UniqueName: \"kubernetes.io/projected/003164c4-6d1b-434e-880f-32860e457de8-kube-api-access-cs5cd\") pod \"kuadrant-operator-catalog-xtkfq\" (UID: \"003164c4-6d1b-434e-880f-32860e457de8\") " pod="kuadrant-system/kuadrant-operator-catalog-xtkfq" Jul 13 14:27:35.354188 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.354147 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xtkfq" Jul 13 14:27:35.390565 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.390531 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xtkfq"] Jul 13 14:27:35.487019 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.486997 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xtkfq"] Jul 13 14:27:35.488903 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:27:35.488871 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003164c4_6d1b_434e_880f_32860e457de8.slice/crio-8afd6246cb2555671f2a991257b1c87728ef6955f2255e15ab8972d382201253 WatchSource:0}: Error finding container 8afd6246cb2555671f2a991257b1c87728ef6955f2255e15ab8972d382201253: Status 404 returned error can't find the container with id 8afd6246cb2555671f2a991257b1c87728ef6955f2255e15ab8972d382201253 Jul 13 14:27:35.608449 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.608419 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ljg5s"] Jul 13 14:27:35.610999 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.610984 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ljg5s" Jul 13 14:27:35.626591 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.626570 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ljg5s"] Jul 13 14:27:35.724852 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.724819 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjbtj\" (UniqueName: \"kubernetes.io/projected/72587244-7f2b-4a79-869e-445dd6ad500a-kube-api-access-kjbtj\") pod \"kuadrant-operator-catalog-ljg5s\" (UID: \"72587244-7f2b-4a79-869e-445dd6ad500a\") " pod="kuadrant-system/kuadrant-operator-catalog-ljg5s" Jul 13 14:27:35.825518 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.825495 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjbtj\" (UniqueName: \"kubernetes.io/projected/72587244-7f2b-4a79-869e-445dd6ad500a-kube-api-access-kjbtj\") pod \"kuadrant-operator-catalog-ljg5s\" (UID: \"72587244-7f2b-4a79-869e-445dd6ad500a\") " pod="kuadrant-system/kuadrant-operator-catalog-ljg5s" Jul 13 14:27:35.836205 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.836181 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjbtj\" (UniqueName: \"kubernetes.io/projected/72587244-7f2b-4a79-869e-445dd6ad500a-kube-api-access-kjbtj\") pod \"kuadrant-operator-catalog-ljg5s\" (UID: \"72587244-7f2b-4a79-869e-445dd6ad500a\") " pod="kuadrant-system/kuadrant-operator-catalog-ljg5s" Jul 13 14:27:35.919802 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:35.919750 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ljg5s" Jul 13 14:27:36.025457 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:36.025423 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:27:36.065440 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:27:36.065408 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72587244_7f2b_4a79_869e_445dd6ad500a.slice/crio-2b797b3ce244316b7cd233d94ea561d088eb390f84bd68e8770aa4cce16ae1ed WatchSource:0}: Error finding container 2b797b3ce244316b7cd233d94ea561d088eb390f84bd68e8770aa4cce16ae1ed: Status 404 returned error can't find the container with id 2b797b3ce244316b7cd233d94ea561d088eb390f84bd68e8770aa4cce16ae1ed Jul 13 14:27:36.071941 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:36.071912 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ljg5s"] Jul 13 14:27:36.120814 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:36.120703 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:27:36.394629 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:36.394561 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xtkfq" event={"ID":"003164c4-6d1b-434e-880f-32860e457de8","Type":"ContainerStarted","Data":"8afd6246cb2555671f2a991257b1c87728ef6955f2255e15ab8972d382201253"} Jul 13 14:27:36.395640 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:36.395617 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ljg5s" event={"ID":"72587244-7f2b-4a79-869e-445dd6ad500a","Type":"ContainerStarted","Data":"2b797b3ce244316b7cd233d94ea561d088eb390f84bd68e8770aa4cce16ae1ed"} Jul 13 14:27:38.406886 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:38.406846 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xtkfq" event={"ID":"003164c4-6d1b-434e-880f-32860e457de8","Type":"ContainerStarted","Data":"d067c99e1161557b6c250f296f77d922ca3734f4c0bdb89ab4368bd0a3ea9bd0"} Jul 13 14:27:38.407305 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:38.406910 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-xtkfq" podUID="003164c4-6d1b-434e-880f-32860e457de8" containerName="registry-server" containerID="cri-o://d067c99e1161557b6c250f296f77d922ca3734f4c0bdb89ab4368bd0a3ea9bd0" gracePeriod=2 Jul 13 14:27:38.433121 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:38.433075 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-xtkfq" podStartSLOduration=1.24621879 podStartE2EDuration="3.433060366s" totalImagesPullingTime="2.186841576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:27:38.431353473 +0000 UTC m=+400.115773800" watchObservedRunningTime="2026-07-13 14:27:38.433060366 +0000 UTC m=+400.117480672" Jul 13 14:27:38.645045 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:38.645020 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xtkfq" Jul 13 14:27:38.748613 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:38.748546 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cs5cd\" (UniqueName: \"kubernetes.io/projected/003164c4-6d1b-434e-880f-32860e457de8-kube-api-access-cs5cd\") pod \"003164c4-6d1b-434e-880f-32860e457de8\" (UID: \"003164c4-6d1b-434e-880f-32860e457de8\") " Jul 13 14:27:38.750620 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:38.750588 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003164c4-6d1b-434e-880f-32860e457de8-kube-api-access-cs5cd" (OuterVolumeSpecName: "kube-api-access-cs5cd") pod "003164c4-6d1b-434e-880f-32860e457de8" (UID: "003164c4-6d1b-434e-880f-32860e457de8"). InnerVolumeSpecName "kube-api-access-cs5cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:27:38.849628 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:38.849605 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs5cd\" (UniqueName: \"kubernetes.io/projected/003164c4-6d1b-434e-880f-32860e457de8-kube-api-access-cs5cd\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:27:39.412517 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:39.412478 2566 generic.go:356] "Generic (PLEG): container finished" podID="003164c4-6d1b-434e-880f-32860e457de8" containerID="d067c99e1161557b6c250f296f77d922ca3734f4c0bdb89ab4368bd0a3ea9bd0" exitCode=0 Jul 13 14:27:39.412937 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:39.412524 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xtkfq" Jul 13 14:27:39.412937 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:39.412560 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xtkfq" event={"ID":"003164c4-6d1b-434e-880f-32860e457de8","Type":"ContainerDied","Data":"d067c99e1161557b6c250f296f77d922ca3734f4c0bdb89ab4368bd0a3ea9bd0"} Jul 13 14:27:39.412937 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:39.412607 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xtkfq" event={"ID":"003164c4-6d1b-434e-880f-32860e457de8","Type":"ContainerDied","Data":"8afd6246cb2555671f2a991257b1c87728ef6955f2255e15ab8972d382201253"} Jul 13 14:27:39.412937 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:39.412627 2566 scope.go:117] "RemoveContainer" containerID="d067c99e1161557b6c250f296f77d922ca3734f4c0bdb89ab4368bd0a3ea9bd0" Jul 13 14:27:39.414150 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:39.414130 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ljg5s" event={"ID":"72587244-7f2b-4a79-869e-445dd6ad500a","Type":"ContainerStarted","Data":"7e1741848d996190921c3cd7334097ad22d5fdffa24088303e13fcd47334be2e"} Jul 13 14:27:39.421190 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:39.421175 2566 scope.go:117] "RemoveContainer" containerID="d067c99e1161557b6c250f296f77d922ca3734f4c0bdb89ab4368bd0a3ea9bd0" Jul 13 14:27:39.421436 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:27:39.421417 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d067c99e1161557b6c250f296f77d922ca3734f4c0bdb89ab4368bd0a3ea9bd0\": container with ID starting with d067c99e1161557b6c250f296f77d922ca3734f4c0bdb89ab4368bd0a3ea9bd0 not found: ID does not exist" containerID="d067c99e1161557b6c250f296f77d922ca3734f4c0bdb89ab4368bd0a3ea9bd0" Jul 13 14:27:39.421476 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:39.421446 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d067c99e1161557b6c250f296f77d922ca3734f4c0bdb89ab4368bd0a3ea9bd0"} err="failed to get container status \"d067c99e1161557b6c250f296f77d922ca3734f4c0bdb89ab4368bd0a3ea9bd0\": rpc error: code = NotFound desc = could not find container \"d067c99e1161557b6c250f296f77d922ca3734f4c0bdb89ab4368bd0a3ea9bd0\": container with ID starting with d067c99e1161557b6c250f296f77d922ca3734f4c0bdb89ab4368bd0a3ea9bd0 not found: ID does not exist" Jul 13 14:27:39.431263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:39.431241 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xtkfq"] Jul 13 14:27:39.436528 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:39.436509 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xtkfq"] Jul 13 14:27:39.454596 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:39.454559 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-ljg5s" podStartSLOduration=1.739505199 podStartE2EDuration="4.454546549s" totalImagesPullingTime="2.71504135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:27:39.452844844 +0000 UTC m=+401.137265151" watchObservedRunningTime="2026-07-13 14:27:39.454546549 +0000 UTC m=+401.138966864" Jul 13 14:27:40.866411 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:40.866380 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="003164c4-6d1b-434e-880f-32860e457de8" path="/var/lib/kubelet/pods/003164c4-6d1b-434e-880f-32860e457de8/volumes" Jul 13 14:27:45.920109 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:45.920071 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-ljg5s" Jul 13 14:27:45.920502 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:45.920143 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-ljg5s" Jul 13 14:27:45.942775 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:45.942746 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-ljg5s" Jul 13 14:27:46.466464 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:46.466437 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-ljg5s" Jul 13 14:27:47.020426 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.020396 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-784fc7cc58-9x5r8"] Jul 13 14:27:47.020769 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.020674 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003164c4-6d1b-434e-880f-32860e457de8" containerName="registry-server" Jul 13 14:27:47.020769 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.020683 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="003164c4-6d1b-434e-880f-32860e457de8" containerName="registry-server" Jul 13 14:27:47.020769 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.020738 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="003164c4-6d1b-434e-880f-32860e457de8" containerName="registry-server" Jul 13 14:27:47.024945 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.024929 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.034877 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.034855 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 13 14:27:47.035970 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.035952 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d5w6n\"" Jul 13 14:27:47.036139 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.035970 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 13 14:27:47.036139 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.035995 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 13 14:27:47.036139 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.035973 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 13 14:27:47.036247 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.036227 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 13 14:27:47.036373 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.036360 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 13 14:27:47.036415 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.036363 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 13 14:27:47.043273 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.043251 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-784fc7cc58-9x5r8"] Jul 13 14:27:47.045080 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.045059 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 13 14:27:47.114473 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.114433 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-oauth-serving-cert\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.114607 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.114487 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-console-oauth-config\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.114607 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.114508 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-console-serving-cert\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.114607 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.114548 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-service-ca\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.114607 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.114568 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-console-config\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.114737 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.114633 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-trusted-ca-bundle\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.114737 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.114669 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5dg5\" (UniqueName: \"kubernetes.io/projected/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-kube-api-access-s5dg5\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.125233 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.125195 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:27:47.215759 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.215733 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-service-ca\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.215872 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.215762 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-console-config\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.215872 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.215812 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-trusted-ca-bundle\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.215872 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.215836 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5dg5\" (UniqueName: \"kubernetes.io/projected/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-kube-api-access-s5dg5\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.216016 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.215876 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-oauth-serving-cert\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.216016 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.215906 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-console-oauth-config\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.216111 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.216038 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-console-serving-cert\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.216569 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.216548 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-service-ca\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.216663 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.216548 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-console-config\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.216752 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.216728 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-oauth-serving-cert\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.216810 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.216768 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-trusted-ca-bundle\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.218306 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.218282 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-console-serving-cert\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.218306 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.218301 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-console-oauth-config\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.232863 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.232844 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5dg5\" (UniqueName: \"kubernetes.io/projected/c0aa2379-1c20-46a0-b1ae-0e45b45f2876-kube-api-access-s5dg5\") pod \"console-784fc7cc58-9x5r8\" (UID: \"c0aa2379-1c20-46a0-b1ae-0e45b45f2876\") " pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.288239 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.288168 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:27:47.333625 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.333597 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:47.463063 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:47.463020 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-784fc7cc58-9x5r8"] Jul 13 14:27:47.465233 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:27:47.465204 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0aa2379_1c20_46a0_b1ae_0e45b45f2876.slice/crio-de59be3d0170d5c232a40bdc91a97b020005d6507ce922b01de624789b3ad4bf WatchSource:0}: Error finding container de59be3d0170d5c232a40bdc91a97b020005d6507ce922b01de624789b3ad4bf: Status 404 returned error can't find the container with id de59be3d0170d5c232a40bdc91a97b020005d6507ce922b01de624789b3ad4bf Jul 13 14:27:48.456732 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:48.456699 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784fc7cc58-9x5r8" event={"ID":"c0aa2379-1c20-46a0-b1ae-0e45b45f2876","Type":"ContainerStarted","Data":"9c4b0657aa7378b1dab70d3ac580c0feb1c520e196fed6c5085500767231d374"} Jul 13 14:27:48.457142 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:48.456746 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784fc7cc58-9x5r8" event={"ID":"c0aa2379-1c20-46a0-b1ae-0e45b45f2876","Type":"ContainerStarted","Data":"de59be3d0170d5c232a40bdc91a97b020005d6507ce922b01de624789b3ad4bf"} Jul 13 14:27:48.485215 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:48.485166 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-784fc7cc58-9x5r8" podStartSLOduration=2.485150331 podStartE2EDuration="2.485150331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:27:48.484910982 +0000 UTC m=+410.169331288" watchObservedRunningTime="2026-07-13 14:27:48.485150331 +0000 UTC m=+410.169570636" Jul 13 14:27:50.638745 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:50.638710 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk"] Jul 13 14:27:50.641739 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:50.641724 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" Jul 13 14:27:50.644670 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:50.644651 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fn6p6\"" Jul 13 14:27:50.653565 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:50.653546 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk"] Jul 13 14:27:50.746017 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:50.745971 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rds5\" (UniqueName: \"kubernetes.io/projected/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-kube-api-access-7rds5\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk\" (UID: \"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" Jul 13 14:27:50.746017 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:50.746024 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk\" (UID: \"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" Jul 13 14:27:50.746226 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:50.746094 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk\" (UID: \"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" Jul 13 14:27:50.847368 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:50.847332 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk\" (UID: \"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" Jul 13 14:27:50.847522 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:50.847392 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rds5\" (UniqueName: \"kubernetes.io/projected/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-kube-api-access-7rds5\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk\" (UID: \"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" Jul 13 14:27:50.847522 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:50.847421 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk\" (UID: \"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" Jul 13 14:27:50.847726 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:50.847705 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk\" (UID: \"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" Jul 13 14:27:50.847766 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:50.847726 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk\" (UID: \"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" Jul 13 14:27:50.858363 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:50.858341 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rds5\" (UniqueName: \"kubernetes.io/projected/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-kube-api-access-7rds5\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk\" (UID: \"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" Jul 13 14:27:50.951368 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:50.951297 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" Jul 13 14:27:51.080151 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.080128 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk"] Jul 13 14:27:51.081920 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:27:51.081894 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ab9bb8_c7c4_49df_8cf7_f9bcaf8e145e.slice/crio-71ca6d415fca03676cd65fdce2b213b1986c835ee19d480119995984f78429aa WatchSource:0}: Error finding container 71ca6d415fca03676cd65fdce2b213b1986c835ee19d480119995984f78429aa: Status 404 returned error can't find the container with id 71ca6d415fca03676cd65fdce2b213b1986c835ee19d480119995984f78429aa Jul 13 14:27:51.251455 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.251388 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw"] Jul 13 14:27:51.254637 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.254621 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" Jul 13 14:27:51.264614 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.264594 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw"] Jul 13 14:27:51.351802 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.351756 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdhqf\" (UniqueName: \"kubernetes.io/projected/ff84688b-cb54-437d-b0d3-6204163e4719-kube-api-access-cdhqf\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw\" (UID: \"ff84688b-cb54-437d-b0d3-6204163e4719\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" Jul 13 14:27:51.351962 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.351874 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff84688b-cb54-437d-b0d3-6204163e4719-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw\" (UID: \"ff84688b-cb54-437d-b0d3-6204163e4719\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" Jul 13 14:27:51.351962 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.351931 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff84688b-cb54-437d-b0d3-6204163e4719-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw\" (UID: \"ff84688b-cb54-437d-b0d3-6204163e4719\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" Jul 13 14:27:51.453020 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.452992 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff84688b-cb54-437d-b0d3-6204163e4719-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw\" (UID: \"ff84688b-cb54-437d-b0d3-6204163e4719\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" Jul 13 14:27:51.453020 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.453020 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff84688b-cb54-437d-b0d3-6204163e4719-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw\" (UID: \"ff84688b-cb54-437d-b0d3-6204163e4719\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" Jul 13 14:27:51.453188 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.453057 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdhqf\" (UniqueName: \"kubernetes.io/projected/ff84688b-cb54-437d-b0d3-6204163e4719-kube-api-access-cdhqf\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw\" (UID: \"ff84688b-cb54-437d-b0d3-6204163e4719\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" Jul 13 14:27:51.453331 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.453313 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff84688b-cb54-437d-b0d3-6204163e4719-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw\" (UID: \"ff84688b-cb54-437d-b0d3-6204163e4719\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" Jul 13 14:27:51.453385 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.453365 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff84688b-cb54-437d-b0d3-6204163e4719-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw\" (UID: \"ff84688b-cb54-437d-b0d3-6204163e4719\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" Jul 13 14:27:51.463598 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.463580 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdhqf\" (UniqueName: \"kubernetes.io/projected/ff84688b-cb54-437d-b0d3-6204163e4719-kube-api-access-cdhqf\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw\" (UID: \"ff84688b-cb54-437d-b0d3-6204163e4719\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" Jul 13 14:27:51.471463 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.471433 2566 generic.go:356] "Generic (PLEG): container finished" podID="19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e" containerID="e23fdeb4cb4d6412e9b719c0499e4382c3d9fa87e7fa89069dfd2bf917d15197" exitCode=0 Jul 13 14:27:51.471572 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.471469 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" event={"ID":"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e","Type":"ContainerDied","Data":"e23fdeb4cb4d6412e9b719c0499e4382c3d9fa87e7fa89069dfd2bf917d15197"} Jul 13 14:27:51.471572 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.471511 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" event={"ID":"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e","Type":"ContainerStarted","Data":"71ca6d415fca03676cd65fdce2b213b1986c835ee19d480119995984f78429aa"} Jul 13 14:27:51.563390 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.563329 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" Jul 13 14:27:51.691440 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.691413 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw"] Jul 13 14:27:51.693411 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:27:51.693384 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff84688b_cb54_437d_b0d3_6204163e4719.slice/crio-4718e8b61de6bdcd7c93922ec21fc965b76b4b55ac28ac4c51f987e3e7f19c85 WatchSource:0}: Error finding container 4718e8b61de6bdcd7c93922ec21fc965b76b4b55ac28ac4c51f987e3e7f19c85: Status 404 returned error can't find the container with id 4718e8b61de6bdcd7c93922ec21fc965b76b4b55ac28ac4c51f987e3e7f19c85 Jul 13 14:27:51.842858 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.842831 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2"] Jul 13 14:27:51.846148 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.846133 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" Jul 13 14:27:51.857087 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.857065 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2"] Jul 13 14:27:51.956158 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.956128 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2\" (UID: \"8a0f93a6-34cc-4645-b6ce-c2d6480527ed\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" Jul 13 14:27:51.956291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.956178 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qwlz\" (UniqueName: \"kubernetes.io/projected/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-kube-api-access-4qwlz\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2\" (UID: \"8a0f93a6-34cc-4645-b6ce-c2d6480527ed\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" Jul 13 14:27:51.956291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:51.956241 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2\" (UID: \"8a0f93a6-34cc-4645-b6ce-c2d6480527ed\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" Jul 13 14:27:52.057022 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.056989 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2\" (UID: \"8a0f93a6-34cc-4645-b6ce-c2d6480527ed\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" Jul 13 14:27:52.057153 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.057028 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2\" (UID: \"8a0f93a6-34cc-4645-b6ce-c2d6480527ed\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" Jul 13 14:27:52.057153 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.057059 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qwlz\" (UniqueName: \"kubernetes.io/projected/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-kube-api-access-4qwlz\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2\" (UID: \"8a0f93a6-34cc-4645-b6ce-c2d6480527ed\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" Jul 13 14:27:52.057367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.057348 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2\" (UID: \"8a0f93a6-34cc-4645-b6ce-c2d6480527ed\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" Jul 13 14:27:52.057402 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.057379 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2\" (UID: \"8a0f93a6-34cc-4645-b6ce-c2d6480527ed\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" Jul 13 14:27:52.068452 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.068430 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qwlz\" (UniqueName: \"kubernetes.io/projected/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-kube-api-access-4qwlz\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2\" (UID: \"8a0f93a6-34cc-4645-b6ce-c2d6480527ed\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" Jul 13 14:27:52.155250 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.155189 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" Jul 13 14:27:52.240340 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.240314 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k"] Jul 13 14:27:52.248649 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.247722 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" Jul 13 14:27:52.255140 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.255114 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k"] Jul 13 14:27:52.359344 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.359314 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fwwb\" (UniqueName: \"kubernetes.io/projected/4b56ed62-746d-4647-bdb3-d5f690201e90-kube-api-access-8fwwb\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k\" (UID: \"4b56ed62-746d-4647-bdb3-d5f690201e90\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" Jul 13 14:27:52.359496 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.359366 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b56ed62-746d-4647-bdb3-d5f690201e90-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k\" (UID: \"4b56ed62-746d-4647-bdb3-d5f690201e90\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" Jul 13 14:27:52.359496 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.359487 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b56ed62-746d-4647-bdb3-d5f690201e90-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k\" (UID: \"4b56ed62-746d-4647-bdb3-d5f690201e90\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" Jul 13 14:27:52.370463 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.370440 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2"] Jul 13 14:27:52.372424 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:27:52.372399 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0f93a6_34cc_4645_b6ce_c2d6480527ed.slice/crio-3148370c0ee7aaa015b3f079d8e3a4d08caea4b31248b022d7482d948c822445 WatchSource:0}: Error finding container 3148370c0ee7aaa015b3f079d8e3a4d08caea4b31248b022d7482d948c822445: Status 404 returned error can't find the container with id 3148370c0ee7aaa015b3f079d8e3a4d08caea4b31248b022d7482d948c822445 Jul 13 14:27:52.460369 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.460333 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b56ed62-746d-4647-bdb3-d5f690201e90-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k\" (UID: \"4b56ed62-746d-4647-bdb3-d5f690201e90\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" Jul 13 14:27:52.460539 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.460406 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b56ed62-746d-4647-bdb3-d5f690201e90-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k\" (UID: \"4b56ed62-746d-4647-bdb3-d5f690201e90\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" Jul 13 14:27:52.460539 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.460461 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fwwb\" (UniqueName: \"kubernetes.io/projected/4b56ed62-746d-4647-bdb3-d5f690201e90-kube-api-access-8fwwb\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k\" (UID: \"4b56ed62-746d-4647-bdb3-d5f690201e90\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" Jul 13 14:27:52.460730 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.460706 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b56ed62-746d-4647-bdb3-d5f690201e90-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k\" (UID: \"4b56ed62-746d-4647-bdb3-d5f690201e90\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" Jul 13 14:27:52.460834 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.460812 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b56ed62-746d-4647-bdb3-d5f690201e90-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k\" (UID: \"4b56ed62-746d-4647-bdb3-d5f690201e90\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" Jul 13 14:27:52.471032 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.471009 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fwwb\" (UniqueName: \"kubernetes.io/projected/4b56ed62-746d-4647-bdb3-d5f690201e90-kube-api-access-8fwwb\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k\" (UID: \"4b56ed62-746d-4647-bdb3-d5f690201e90\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" Jul 13 14:27:52.477982 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.477953 2566 generic.go:356] "Generic (PLEG): container finished" podID="ff84688b-cb54-437d-b0d3-6204163e4719" containerID="471a1a2a0a9e324cc82173fe291c914ce1f2ec80e4ca2f3659644ca9eecb2a34" exitCode=0 Jul 13 14:27:52.478096 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.477990 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" event={"ID":"ff84688b-cb54-437d-b0d3-6204163e4719","Type":"ContainerDied","Data":"471a1a2a0a9e324cc82173fe291c914ce1f2ec80e4ca2f3659644ca9eecb2a34"} Jul 13 14:27:52.478096 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.478055 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" event={"ID":"ff84688b-cb54-437d-b0d3-6204163e4719","Type":"ContainerStarted","Data":"4718e8b61de6bdcd7c93922ec21fc965b76b4b55ac28ac4c51f987e3e7f19c85"} Jul 13 14:27:52.479924 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.479887 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" event={"ID":"8a0f93a6-34cc-4645-b6ce-c2d6480527ed","Type":"ContainerStarted","Data":"c5fbd7ef123e59431fc7ea43fb55ed36bdcd4e2c5ff31b57ab61718be3f78895"} Jul 13 14:27:52.480032 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.479942 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" event={"ID":"8a0f93a6-34cc-4645-b6ce-c2d6480527ed","Type":"ContainerStarted","Data":"3148370c0ee7aaa015b3f079d8e3a4d08caea4b31248b022d7482d948c822445"} Jul 13 14:27:52.563593 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.563566 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" Jul 13 14:27:52.711553 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:52.711527 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k"] Jul 13 14:27:52.726632 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:27:52.726600 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b56ed62_746d_4647_bdb3_d5f690201e90.slice/crio-75d3edbfcdeace748137bb4b4e85929e34db339dc07e61f6970e9a620399390d WatchSource:0}: Error finding container 75d3edbfcdeace748137bb4b4e85929e34db339dc07e61f6970e9a620399390d: Status 404 returned error can't find the container with id 75d3edbfcdeace748137bb4b4e85929e34db339dc07e61f6970e9a620399390d Jul 13 14:27:53.485816 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:53.485767 2566 generic.go:356] "Generic (PLEG): container finished" podID="8a0f93a6-34cc-4645-b6ce-c2d6480527ed" containerID="c5fbd7ef123e59431fc7ea43fb55ed36bdcd4e2c5ff31b57ab61718be3f78895" exitCode=0 Jul 13 14:27:53.485950 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:53.485855 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" event={"ID":"8a0f93a6-34cc-4645-b6ce-c2d6480527ed","Type":"ContainerDied","Data":"c5fbd7ef123e59431fc7ea43fb55ed36bdcd4e2c5ff31b57ab61718be3f78895"} Jul 13 14:27:53.487421 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:53.487391 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b56ed62-746d-4647-bdb3-d5f690201e90" containerID="1dbd0beed80884675f75519635dc9d7f90dc69962832f85a8ac6b336eff631a2" exitCode=0 Jul 13 14:27:53.487515 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:53.487430 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" event={"ID":"4b56ed62-746d-4647-bdb3-d5f690201e90","Type":"ContainerDied","Data":"1dbd0beed80884675f75519635dc9d7f90dc69962832f85a8ac6b336eff631a2"} Jul 13 14:27:53.487515 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:53.487474 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" event={"ID":"4b56ed62-746d-4647-bdb3-d5f690201e90","Type":"ContainerStarted","Data":"75d3edbfcdeace748137bb4b4e85929e34db339dc07e61f6970e9a620399390d"} Jul 13 14:27:53.489418 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:53.489309 2566 generic.go:356] "Generic (PLEG): container finished" podID="19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e" containerID="ddbaa006889c7cd7de483b80e03a52e76950c6fc0d9f4dd419c288f8f7ae67c9" exitCode=0 Jul 13 14:27:53.489418 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:53.489378 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" event={"ID":"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e","Type":"ContainerDied","Data":"ddbaa006889c7cd7de483b80e03a52e76950c6fc0d9f4dd419c288f8f7ae67c9"} Jul 13 14:27:53.491406 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:53.491385 2566 generic.go:356] "Generic (PLEG): container finished" podID="ff84688b-cb54-437d-b0d3-6204163e4719" containerID="c49ea59edec181b818d35e5c7bdf610870035c1c7d0a761dabd6051f4f052ede" exitCode=0 Jul 13 14:27:53.491494 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:53.491444 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" event={"ID":"ff84688b-cb54-437d-b0d3-6204163e4719","Type":"ContainerDied","Data":"c49ea59edec181b818d35e5c7bdf610870035c1c7d0a761dabd6051f4f052ede"} Jul 13 14:27:54.499171 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:54.499147 2566 generic.go:356] "Generic (PLEG): container finished" podID="ff84688b-cb54-437d-b0d3-6204163e4719" containerID="f9f8928d757cd451d8e18aaf9a243e7f1cc02a4d19b5f1770db7bd62455be59d" exitCode=0 Jul 13 14:27:54.499498 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:54.499213 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" event={"ID":"ff84688b-cb54-437d-b0d3-6204163e4719","Type":"ContainerDied","Data":"f9f8928d757cd451d8e18aaf9a243e7f1cc02a4d19b5f1770db7bd62455be59d"} Jul 13 14:27:54.501215 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:54.501185 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" event={"ID":"8a0f93a6-34cc-4645-b6ce-c2d6480527ed","Type":"ContainerStarted","Data":"9fdb71fd930906c877e9551642dc1161a1a906c9872c8af85a2b4e7d5785bcd5"} Jul 13 14:27:54.503173 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:54.503149 2566 generic.go:356] "Generic (PLEG): container finished" podID="19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e" containerID="2f5692c536d3b5bb5c25da35c8e91d04775ad795555b5d03317402f1f9f92b6a" exitCode=0 Jul 13 14:27:54.503258 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:54.503222 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" event={"ID":"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e","Type":"ContainerDied","Data":"2f5692c536d3b5bb5c25da35c8e91d04775ad795555b5d03317402f1f9f92b6a"} Jul 13 14:27:55.509041 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.509005 2566 generic.go:356] "Generic (PLEG): container finished" podID="8a0f93a6-34cc-4645-b6ce-c2d6480527ed" containerID="9fdb71fd930906c877e9551642dc1161a1a906c9872c8af85a2b4e7d5785bcd5" exitCode=0 Jul 13 14:27:55.509468 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.509084 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" event={"ID":"8a0f93a6-34cc-4645-b6ce-c2d6480527ed","Type":"ContainerDied","Data":"9fdb71fd930906c877e9551642dc1161a1a906c9872c8af85a2b4e7d5785bcd5"} Jul 13 14:27:55.510868 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.510847 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b56ed62-746d-4647-bdb3-d5f690201e90" containerID="b238825cc9ad510cbbd692901c5089ab6af4db468466c8abc5a5ea19957c678a" exitCode=0 Jul 13 14:27:55.510964 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.510936 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" event={"ID":"4b56ed62-746d-4647-bdb3-d5f690201e90","Type":"ContainerDied","Data":"b238825cc9ad510cbbd692901c5089ab6af4db468466c8abc5a5ea19957c678a"} Jul 13 14:27:55.662572 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.662546 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" Jul 13 14:27:55.783916 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.783850 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" Jul 13 14:27:55.790263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.790246 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdhqf\" (UniqueName: \"kubernetes.io/projected/ff84688b-cb54-437d-b0d3-6204163e4719-kube-api-access-cdhqf\") pod \"ff84688b-cb54-437d-b0d3-6204163e4719\" (UID: \"ff84688b-cb54-437d-b0d3-6204163e4719\") " Jul 13 14:27:55.790345 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.790277 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff84688b-cb54-437d-b0d3-6204163e4719-bundle\") pod \"ff84688b-cb54-437d-b0d3-6204163e4719\" (UID: \"ff84688b-cb54-437d-b0d3-6204163e4719\") " Jul 13 14:27:55.790345 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.790335 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff84688b-cb54-437d-b0d3-6204163e4719-util\") pod \"ff84688b-cb54-437d-b0d3-6204163e4719\" (UID: \"ff84688b-cb54-437d-b0d3-6204163e4719\") " Jul 13 14:27:55.791062 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.791040 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff84688b-cb54-437d-b0d3-6204163e4719-bundle" (OuterVolumeSpecName: "bundle") pod "ff84688b-cb54-437d-b0d3-6204163e4719" (UID: "ff84688b-cb54-437d-b0d3-6204163e4719"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:27:55.792327 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.792306 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff84688b-cb54-437d-b0d3-6204163e4719-kube-api-access-cdhqf" (OuterVolumeSpecName: "kube-api-access-cdhqf") pod "ff84688b-cb54-437d-b0d3-6204163e4719" (UID: "ff84688b-cb54-437d-b0d3-6204163e4719"). InnerVolumeSpecName "kube-api-access-cdhqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:27:55.795370 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.795350 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff84688b-cb54-437d-b0d3-6204163e4719-util" (OuterVolumeSpecName: "util") pod "ff84688b-cb54-437d-b0d3-6204163e4719" (UID: "ff84688b-cb54-437d-b0d3-6204163e4719"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:27:55.891425 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.891390 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-util\") pod \"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e\" (UID: \"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e\") " Jul 13 14:27:55.891592 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.891445 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-bundle\") pod \"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e\" (UID: \"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e\") " Jul 13 14:27:55.891592 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.891477 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rds5\" (UniqueName: \"kubernetes.io/projected/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-kube-api-access-7rds5\") pod \"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e\" (UID: \"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e\") " Jul 13 14:27:55.891718 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.891610 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdhqf\" (UniqueName: \"kubernetes.io/projected/ff84688b-cb54-437d-b0d3-6204163e4719-kube-api-access-cdhqf\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:27:55.891718 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.891620 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ff84688b-cb54-437d-b0d3-6204163e4719-bundle\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:27:55.891718 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.891628 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ff84688b-cb54-437d-b0d3-6204163e4719-util\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:27:55.891937 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.891910 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-bundle" (OuterVolumeSpecName: "bundle") pod "19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e" (UID: "19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:27:55.893564 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.893538 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-kube-api-access-7rds5" (OuterVolumeSpecName: "kube-api-access-7rds5") pod "19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e" (UID: "19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e"). InnerVolumeSpecName "kube-api-access-7rds5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:27:55.897465 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.897427 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-util" (OuterVolumeSpecName: "util") pod "19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e" (UID: "19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:27:55.992886 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.992857 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-util\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:27:55.992886 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.992883 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-bundle\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:27:55.993071 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:55.992895 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rds5\" (UniqueName: \"kubernetes.io/projected/19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e-kube-api-access-7rds5\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:27:56.517043 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:56.517011 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" Jul 13 14:27:56.517419 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:56.517020 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw" event={"ID":"ff84688b-cb54-437d-b0d3-6204163e4719","Type":"ContainerDied","Data":"4718e8b61de6bdcd7c93922ec21fc965b76b4b55ac28ac4c51f987e3e7f19c85"} Jul 13 14:27:56.517419 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:56.517131 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4718e8b61de6bdcd7c93922ec21fc965b76b4b55ac28ac4c51f987e3e7f19c85" Jul 13 14:27:56.519088 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:56.519065 2566 generic.go:356] "Generic (PLEG): container finished" podID="8a0f93a6-34cc-4645-b6ce-c2d6480527ed" containerID="84dbb2a544dc864dc95e8073969aee03482b4b17b68b58baeaa605e0eb8b02fa" exitCode=0 Jul 13 14:27:56.519218 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:56.519135 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" event={"ID":"8a0f93a6-34cc-4645-b6ce-c2d6480527ed","Type":"ContainerDied","Data":"84dbb2a544dc864dc95e8073969aee03482b4b17b68b58baeaa605e0eb8b02fa"} Jul 13 14:27:56.521166 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:56.521142 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b56ed62-746d-4647-bdb3-d5f690201e90" containerID="adff6826e4333b9f2af547b1c9f63a205ccd039e9a2e678a8cd6fe26970f7356" exitCode=0 Jul 13 14:27:56.521268 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:56.521168 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" event={"ID":"4b56ed62-746d-4647-bdb3-d5f690201e90","Type":"ContainerDied","Data":"adff6826e4333b9f2af547b1c9f63a205ccd039e9a2e678a8cd6fe26970f7356"} Jul 13 14:27:56.522868 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:56.522851 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" event={"ID":"19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e","Type":"ContainerDied","Data":"71ca6d415fca03676cd65fdce2b213b1986c835ee19d480119995984f78429aa"} Jul 13 14:27:56.522934 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:56.522882 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71ca6d415fca03676cd65fdce2b213b1986c835ee19d480119995984f78429aa" Jul 13 14:27:56.522934 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:56.522887 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk" Jul 13 14:27:57.334362 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.334332 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:57.334650 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.334395 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:57.338907 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.338884 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:57.531939 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.531915 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-784fc7cc58-9x5r8" Jul 13 14:27:57.691602 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.691581 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" Jul 13 14:27:57.694071 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.694053 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" Jul 13 14:27:57.719300 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.719278 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:27:57.809579 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.809540 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-bundle\") pod \"8a0f93a6-34cc-4645-b6ce-c2d6480527ed\" (UID: \"8a0f93a6-34cc-4645-b6ce-c2d6480527ed\") " Jul 13 14:27:57.809769 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.809594 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-util\") pod \"8a0f93a6-34cc-4645-b6ce-c2d6480527ed\" (UID: \"8a0f93a6-34cc-4645-b6ce-c2d6480527ed\") " Jul 13 14:27:57.809769 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.809647 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b56ed62-746d-4647-bdb3-d5f690201e90-util\") pod \"4b56ed62-746d-4647-bdb3-d5f690201e90\" (UID: \"4b56ed62-746d-4647-bdb3-d5f690201e90\") " Jul 13 14:27:57.809769 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.809677 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b56ed62-746d-4647-bdb3-d5f690201e90-bundle\") pod \"4b56ed62-746d-4647-bdb3-d5f690201e90\" (UID: \"4b56ed62-746d-4647-bdb3-d5f690201e90\") " Jul 13 14:27:57.809769 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.809729 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qwlz\" (UniqueName: \"kubernetes.io/projected/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-kube-api-access-4qwlz\") pod \"8a0f93a6-34cc-4645-b6ce-c2d6480527ed\" (UID: \"8a0f93a6-34cc-4645-b6ce-c2d6480527ed\") " Jul 13 14:27:57.810025 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.809775 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fwwb\" (UniqueName: \"kubernetes.io/projected/4b56ed62-746d-4647-bdb3-d5f690201e90-kube-api-access-8fwwb\") pod \"4b56ed62-746d-4647-bdb3-d5f690201e90\" (UID: \"4b56ed62-746d-4647-bdb3-d5f690201e90\") " Jul 13 14:27:57.810195 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.810162 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-bundle" (OuterVolumeSpecName: "bundle") pod "8a0f93a6-34cc-4645-b6ce-c2d6480527ed" (UID: "8a0f93a6-34cc-4645-b6ce-c2d6480527ed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:27:57.810364 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.810316 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-bundle\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:27:57.810432 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.810365 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b56ed62-746d-4647-bdb3-d5f690201e90-bundle" (OuterVolumeSpecName: "bundle") pod "4b56ed62-746d-4647-bdb3-d5f690201e90" (UID: "4b56ed62-746d-4647-bdb3-d5f690201e90"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:27:57.811924 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.811898 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b56ed62-746d-4647-bdb3-d5f690201e90-kube-api-access-8fwwb" (OuterVolumeSpecName: "kube-api-access-8fwwb") pod "4b56ed62-746d-4647-bdb3-d5f690201e90" (UID: "4b56ed62-746d-4647-bdb3-d5f690201e90"). InnerVolumeSpecName "kube-api-access-8fwwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:27:57.812016 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.811959 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-kube-api-access-4qwlz" (OuterVolumeSpecName: "kube-api-access-4qwlz") pod "8a0f93a6-34cc-4645-b6ce-c2d6480527ed" (UID: "8a0f93a6-34cc-4645-b6ce-c2d6480527ed"). InnerVolumeSpecName "kube-api-access-4qwlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:27:57.817461 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.817438 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-util" (OuterVolumeSpecName: "util") pod "8a0f93a6-34cc-4645-b6ce-c2d6480527ed" (UID: "8a0f93a6-34cc-4645-b6ce-c2d6480527ed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:27:57.818371 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.818353 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b56ed62-746d-4647-bdb3-d5f690201e90-util" (OuterVolumeSpecName: "util") pod "4b56ed62-746d-4647-bdb3-d5f690201e90" (UID: "4b56ed62-746d-4647-bdb3-d5f690201e90"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:27:57.824372 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.824331 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:27:57.895358 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.895301 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:27:57.912016 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.911989 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qwlz\" (UniqueName: \"kubernetes.io/projected/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-kube-api-access-4qwlz\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:27:57.912107 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.912031 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fwwb\" (UniqueName: \"kubernetes.io/projected/4b56ed62-746d-4647-bdb3-d5f690201e90-kube-api-access-8fwwb\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:27:57.912107 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.912046 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8a0f93a6-34cc-4645-b6ce-c2d6480527ed-util\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:27:57.912107 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.912060 2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4b56ed62-746d-4647-bdb3-d5f690201e90-util\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:27:57.912107 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:57.912072 2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4b56ed62-746d-4647-bdb3-d5f690201e90-bundle\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:27:58.534436 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:58.534407 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" Jul 13 14:27:58.534901 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:58.534409 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2" event={"ID":"8a0f93a6-34cc-4645-b6ce-c2d6480527ed","Type":"ContainerDied","Data":"3148370c0ee7aaa015b3f079d8e3a4d08caea4b31248b022d7482d948c822445"} Jul 13 14:27:58.534901 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:58.534559 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3148370c0ee7aaa015b3f079d8e3a4d08caea4b31248b022d7482d948c822445" Jul 13 14:27:58.536265 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:58.536247 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" Jul 13 14:27:58.536405 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:58.536281 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k" event={"ID":"4b56ed62-746d-4647-bdb3-d5f690201e90","Type":"ContainerDied","Data":"75d3edbfcdeace748137bb4b4e85929e34db339dc07e61f6970e9a620399390d"} Jul 13 14:27:58.536405 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:27:58.536316 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75d3edbfcdeace748137bb4b4e85929e34db339dc07e61f6970e9a620399390d" Jul 13 14:28:04.592903 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:04.592869 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:07.199150 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199118 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx"] Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199391 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0f93a6-34cc-4645-b6ce-c2d6480527ed" containerName="extract" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199401 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0f93a6-34cc-4645-b6ce-c2d6480527ed" containerName="extract" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199413 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff84688b-cb54-437d-b0d3-6204163e4719" containerName="extract" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199419 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff84688b-cb54-437d-b0d3-6204163e4719" containerName="extract" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199430 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b56ed62-746d-4647-bdb3-d5f690201e90" containerName="pull" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199436 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b56ed62-746d-4647-bdb3-d5f690201e90" containerName="pull" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199441 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b56ed62-746d-4647-bdb3-d5f690201e90" containerName="extract" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199446 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b56ed62-746d-4647-bdb3-d5f690201e90" containerName="extract" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199454 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e" containerName="util" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199460 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e" containerName="util" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199465 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b56ed62-746d-4647-bdb3-d5f690201e90" containerName="util" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199470 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b56ed62-746d-4647-bdb3-d5f690201e90" containerName="util" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199476 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0f93a6-34cc-4645-b6ce-c2d6480527ed" containerName="pull" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199480 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0f93a6-34cc-4645-b6ce-c2d6480527ed" containerName="pull" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199486 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff84688b-cb54-437d-b0d3-6204163e4719" containerName="util" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199493 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff84688b-cb54-437d-b0d3-6204163e4719" containerName="util" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199499 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0f93a6-34cc-4645-b6ce-c2d6480527ed" containerName="util" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199504 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0f93a6-34cc-4645-b6ce-c2d6480527ed" containerName="util" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199509 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e" containerName="pull" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199513 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e" containerName="pull" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199520 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e" containerName="extract" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199526 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e" containerName="extract" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199532 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff84688b-cb54-437d-b0d3-6204163e4719" containerName="pull" Jul 13 14:28:07.199529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199537 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff84688b-cb54-437d-b0d3-6204163e4719" containerName="pull" Jul 13 14:28:07.200358 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199591 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e" containerName="extract" Jul 13 14:28:07.200358 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199599 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a0f93a6-34cc-4645-b6ce-c2d6480527ed" containerName="extract" Jul 13 14:28:07.200358 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199604 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b56ed62-746d-4647-bdb3-d5f690201e90" containerName="extract" Jul 13 14:28:07.200358 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.199610 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff84688b-cb54-437d-b0d3-6204163e4719" containerName="extract" Jul 13 14:28:07.204471 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.204455 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" Jul 13 14:28:07.208644 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.208621 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-95zgz\"" Jul 13 14:28:07.215778 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.215757 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx"] Jul 13 14:28:07.285656 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.285624 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz2nf\" (UniqueName: \"kubernetes.io/projected/91c28b56-8058-45ba-b1b0-1d12036565ae-kube-api-access-zz2nf\") pod \"limitador-operator-controller-manager-859d7b78bd-smhqx\" (UID: \"91c28b56-8058-45ba-b1b0-1d12036565ae\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" Jul 13 14:28:07.386246 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.386214 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz2nf\" (UniqueName: \"kubernetes.io/projected/91c28b56-8058-45ba-b1b0-1d12036565ae-kube-api-access-zz2nf\") pod \"limitador-operator-controller-manager-859d7b78bd-smhqx\" (UID: \"91c28b56-8058-45ba-b1b0-1d12036565ae\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" Jul 13 14:28:07.397069 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.397047 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz2nf\" (UniqueName: \"kubernetes.io/projected/91c28b56-8058-45ba-b1b0-1d12036565ae-kube-api-access-zz2nf\") pod \"limitador-operator-controller-manager-859d7b78bd-smhqx\" (UID: \"91c28b56-8058-45ba-b1b0-1d12036565ae\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" Jul 13 14:28:07.514238 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.514167 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" Jul 13 14:28:07.648967 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.644562 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx"] Jul 13 14:28:07.650175 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:28:07.650143 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c28b56_8058_45ba_b1b0_1d12036565ae.slice/crio-831e214348f815a1836c8ac3653821f189ac3badbfeb16d34472005f8f6270d0 WatchSource:0}: Error finding container 831e214348f815a1836c8ac3653821f189ac3badbfeb16d34472005f8f6270d0: Status 404 returned error can't find the container with id 831e214348f815a1836c8ac3653821f189ac3badbfeb16d34472005f8f6270d0 Jul 13 14:28:07.936134 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:07.936101 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:08.582038 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:08.582002 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" event={"ID":"91c28b56-8058-45ba-b1b0-1d12036565ae","Type":"ContainerStarted","Data":"831e214348f815a1836c8ac3653821f189ac3badbfeb16d34472005f8f6270d0"} Jul 13 14:28:08.781835 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:08.781801 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:10.335868 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:10.335832 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:10.593071 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:10.592985 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" event={"ID":"91c28b56-8058-45ba-b1b0-1d12036565ae","Type":"ContainerStarted","Data":"6cff8f9922f9424987048d2c83a3e52d1a75fcb41e273240292823a16f963794"} Jul 13 14:28:10.593237 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:10.593079 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" Jul 13 14:28:10.618594 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:10.618546 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" podStartSLOduration=1.6816095180000001 podStartE2EDuration="3.618529101s" totalImagesPullingTime="1.936919583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:28:10.616444574 +0000 UTC m=+432.300864894" watchObservedRunningTime="2026-07-13 14:28:10.618529101 +0000 UTC m=+432.302949407" Jul 13 14:28:11.262308 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:11.262275 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hks7l"] Jul 13 14:28:11.265600 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:11.265583 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hks7l" Jul 13 14:28:11.268316 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:11.268297 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bbm8h\"" Jul 13 14:28:11.279439 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:11.279411 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hks7l"] Jul 13 14:28:11.422272 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:11.422238 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4r4c\" (UniqueName: \"kubernetes.io/projected/0a48aa30-c211-4289-bbc5-edeb0d05d537-kube-api-access-p4r4c\") pod \"authorino-operator-75cc8d5cb7-hks7l\" (UID: \"0a48aa30-c211-4289-bbc5-edeb0d05d537\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hks7l" Jul 13 14:28:11.523657 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:11.523590 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4r4c\" (UniqueName: \"kubernetes.io/projected/0a48aa30-c211-4289-bbc5-edeb0d05d537-kube-api-access-p4r4c\") pod \"authorino-operator-75cc8d5cb7-hks7l\" (UID: \"0a48aa30-c211-4289-bbc5-edeb0d05d537\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hks7l" Jul 13 14:28:11.539995 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:11.539969 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4r4c\" (UniqueName: \"kubernetes.io/projected/0a48aa30-c211-4289-bbc5-edeb0d05d537-kube-api-access-p4r4c\") pod \"authorino-operator-75cc8d5cb7-hks7l\" (UID: \"0a48aa30-c211-4289-bbc5-edeb0d05d537\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-hks7l" Jul 13 14:28:11.574833 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:11.574778 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hks7l" Jul 13 14:28:11.709765 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:11.709737 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-hks7l"] Jul 13 14:28:11.712631 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:28:11.712601 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a48aa30_c211_4289_bbc5_edeb0d05d537.slice/crio-307c883436f8b6ba09b2529827370f2dac053e7109787ae654877ea945d0034f WatchSource:0}: Error finding container 307c883436f8b6ba09b2529827370f2dac053e7109787ae654877ea945d0034f: Status 404 returned error can't find the container with id 307c883436f8b6ba09b2529827370f2dac053e7109787ae654877ea945d0034f Jul 13 14:28:12.029012 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:12.028980 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:12.341965 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:12.341940 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:12.605279 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:12.605197 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hks7l" event={"ID":"0a48aa30-c211-4289-bbc5-edeb0d05d537","Type":"ContainerStarted","Data":"307c883436f8b6ba09b2529827370f2dac053e7109787ae654877ea945d0034f"} Jul 13 14:28:14.007575 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:14.007540 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:14.571894 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:14.571865 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-t5p2z"] Jul 13 14:28:14.575214 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:14.575196 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t5p2z" Jul 13 14:28:14.580546 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:14.580524 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 13 14:28:14.580671 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:14.580601 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-s777f\"" Jul 13 14:28:14.592623 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:14.592597 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-t5p2z"] Jul 13 14:28:14.619594 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:14.619558 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hks7l" event={"ID":"0a48aa30-c211-4289-bbc5-edeb0d05d537","Type":"ContainerStarted","Data":"5d283918426cf84c628a99ee6c8fc30a475da04e4e5135a77c375fbb6c133928"} Jul 13 14:28:14.619737 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:14.619710 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hks7l" Jul 13 14:28:14.648816 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:14.648760 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hks7l" podStartSLOduration=1.090440958 podStartE2EDuration="3.648746124s" totalImagesPullingTime="2.558305166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:28:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:28:14.646490493 +0000 UTC m=+436.330910799" watchObservedRunningTime="2026-07-13 14:28:14.648746124 +0000 UTC m=+436.333166430" Jul 13 14:28:14.752266 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:14.752235 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkggj\" (UniqueName: \"kubernetes.io/projected/1fb2a287-7049-45b7-b51a-554483e1ee4b-kube-api-access-kkggj\") pod \"dns-operator-controller-manager-864dd78c4c-t5p2z\" (UID: \"1fb2a287-7049-45b7-b51a-554483e1ee4b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t5p2z" Jul 13 14:28:14.852874 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:14.852779 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkggj\" (UniqueName: \"kubernetes.io/projected/1fb2a287-7049-45b7-b51a-554483e1ee4b-kube-api-access-kkggj\") pod \"dns-operator-controller-manager-864dd78c4c-t5p2z\" (UID: \"1fb2a287-7049-45b7-b51a-554483e1ee4b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t5p2z" Jul 13 14:28:14.862337 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:14.862306 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkggj\" (UniqueName: \"kubernetes.io/projected/1fb2a287-7049-45b7-b51a-554483e1ee4b-kube-api-access-kkggj\") pod \"dns-operator-controller-manager-864dd78c4c-t5p2z\" (UID: \"1fb2a287-7049-45b7-b51a-554483e1ee4b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t5p2z" Jul 13 14:28:14.885397 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:14.885358 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t5p2z" Jul 13 14:28:15.029562 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:15.029528 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-t5p2z"] Jul 13 14:28:15.030169 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:28:15.030143 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb2a287_7049_45b7_b51a_554483e1ee4b.slice/crio-e9c6385480a47ced01602f90149dffb7376e69d08f3bdc6828951806584b9e3e WatchSource:0}: Error finding container e9c6385480a47ced01602f90149dffb7376e69d08f3bdc6828951806584b9e3e: Status 404 returned error can't find the container with id e9c6385480a47ced01602f90149dffb7376e69d08f3bdc6828951806584b9e3e Jul 13 14:28:15.167572 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:15.167496 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:15.261430 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:15.261403 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:15.625735 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:15.625700 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t5p2z" event={"ID":"1fb2a287-7049-45b7-b51a-554483e1ee4b","Type":"ContainerStarted","Data":"e9c6385480a47ced01602f90149dffb7376e69d08f3bdc6828951806584b9e3e"} Jul 13 14:28:16.976654 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:16.976581 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:19.221152 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:19.221112 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p"] Jul 13 14:28:19.224387 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:19.224365 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:19.226939 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:19.226917 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-clhnl\"" Jul 13 14:28:19.238684 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:19.238664 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p"] Jul 13 14:28:19.392469 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:19.392437 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmdrf\" (UniqueName: \"kubernetes.io/projected/cad8c529-921a-4491-863b-4ba4336cf869-kube-api-access-qmdrf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-txh6p\" (UID: \"cad8c529-921a-4491-863b-4ba4336cf869\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:19.392631 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:19.392476 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cad8c529-921a-4491-863b-4ba4336cf869-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-txh6p\" (UID: \"cad8c529-921a-4491-863b-4ba4336cf869\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:19.493296 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:19.493222 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmdrf\" (UniqueName: \"kubernetes.io/projected/cad8c529-921a-4491-863b-4ba4336cf869-kube-api-access-qmdrf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-txh6p\" (UID: \"cad8c529-921a-4491-863b-4ba4336cf869\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:19.493296 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:19.493259 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cad8c529-921a-4491-863b-4ba4336cf869-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-txh6p\" (UID: \"cad8c529-921a-4491-863b-4ba4336cf869\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:19.493631 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:19.493615 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cad8c529-921a-4491-863b-4ba4336cf869-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-txh6p\" (UID: \"cad8c529-921a-4491-863b-4ba4336cf869\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:19.507379 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:19.507348 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmdrf\" (UniqueName: \"kubernetes.io/projected/cad8c529-921a-4491-863b-4ba4336cf869-kube-api-access-qmdrf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-txh6p\" (UID: \"cad8c529-921a-4491-863b-4ba4336cf869\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:19.534200 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:19.534168 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:19.679936 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:19.679912 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p"] Jul 13 14:28:19.682121 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:28:19.682089 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad8c529_921a_4491_863b_4ba4336cf869.slice/crio-b12bd833e67113bd5756d777f520d2ce3ed658e8daef7b241276c8f61506ba44 WatchSource:0}: Error finding container b12bd833e67113bd5756d777f520d2ce3ed658e8daef7b241276c8f61506ba44: Status 404 returned error can't find the container with id b12bd833e67113bd5756d777f520d2ce3ed658e8daef7b241276c8f61506ba44 Jul 13 14:28:20.153936 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:20.153852 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:20.253075 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:20.253047 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:20.652833 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:20.652794 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" event={"ID":"cad8c529-921a-4491-863b-4ba4336cf869","Type":"ContainerStarted","Data":"b12bd833e67113bd5756d777f520d2ce3ed658e8daef7b241276c8f61506ba44"} Jul 13 14:28:21.600796 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:21.600758 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" Jul 13 14:28:21.659571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:21.659540 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t5p2z" event={"ID":"1fb2a287-7049-45b7-b51a-554483e1ee4b","Type":"ContainerStarted","Data":"d05720234e5006c55a130b4ed5cec73cbfa3b5062d2e6911ddf17bde9f6d4f82"} Jul 13 14:28:21.659766 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:21.659658 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t5p2z" Jul 13 14:28:21.684946 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:21.684899 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t5p2z" podStartSLOduration=1.902255042 podStartE2EDuration="7.684884274s" totalImagesPullingTime="5.782629232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:28:21.683330754 +0000 UTC m=+443.367751059" watchObservedRunningTime="2026-07-13 14:28:21.684884274 +0000 UTC m=+443.369304580" Jul 13 14:28:25.554106 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:25.554075 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:25.627669 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:25.627645 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-hks7l" Jul 13 14:28:25.659055 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:25.659025 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:25.947290 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:25.947265 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:27.731263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:27.731234 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:28.694745 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:28.694715 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" event={"ID":"cad8c529-921a-4491-863b-4ba4336cf869","Type":"ContainerStarted","Data":"727654c690ab2e05a49c3415ab406205f899f57b56bb282f8e0b21fa786f34e8"} Jul 13 14:28:28.694902 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:28.694813 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:28.719575 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:28.719531 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" podStartSLOduration=0.86006344 podStartE2EDuration="9.719516504s" totalImagesPullingTime="8.859453064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:28:28.717541837 +0000 UTC m=+450.401962142" watchObservedRunningTime="2026-07-13 14:28:28.719516504 +0000 UTC m=+450.403936810" Jul 13 14:28:28.952133 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:28.952074 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:29.051332 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:29.051305 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:31.330906 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.330867 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr"] Jul 13 14:28:31.333878 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.333853 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:31.334469 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.334454 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr" Jul 13 14:28:31.338253 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.338228 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 13 14:28:31.338354 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.338268 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 13 14:28:31.338354 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.338271 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fn6p6\"" Jul 13 14:28:31.342714 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.342694 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr"] Jul 13 14:28:31.386813 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.386766 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e453e581-c08e-4391-87d1-af26b5a59d45-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-5f2dr\" (UID: \"e453e581-c08e-4391-87d1-af26b5a59d45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr" Jul 13 14:28:31.386963 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.386849 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e453e581-c08e-4391-87d1-af26b5a59d45-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5f2dr\" (UID: \"e453e581-c08e-4391-87d1-af26b5a59d45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr" Jul 13 14:28:31.386963 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.386879 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t5g7\" (UniqueName: \"kubernetes.io/projected/e453e581-c08e-4391-87d1-af26b5a59d45-kube-api-access-8t5g7\") pod \"kuadrant-console-plugin-66b5774985-5f2dr\" (UID: \"e453e581-c08e-4391-87d1-af26b5a59d45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr" Jul 13 14:28:31.396609 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.396576 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:31.487423 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.487397 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e453e581-c08e-4391-87d1-af26b5a59d45-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-5f2dr\" (UID: \"e453e581-c08e-4391-87d1-af26b5a59d45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr" Jul 13 14:28:31.487604 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.487442 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e453e581-c08e-4391-87d1-af26b5a59d45-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5f2dr\" (UID: \"e453e581-c08e-4391-87d1-af26b5a59d45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr" Jul 13 14:28:31.487604 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.487465 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t5g7\" (UniqueName: \"kubernetes.io/projected/e453e581-c08e-4391-87d1-af26b5a59d45-kube-api-access-8t5g7\") pod \"kuadrant-console-plugin-66b5774985-5f2dr\" (UID: \"e453e581-c08e-4391-87d1-af26b5a59d45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr" Jul 13 14:28:31.488137 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.488114 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e453e581-c08e-4391-87d1-af26b5a59d45-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-5f2dr\" (UID: \"e453e581-c08e-4391-87d1-af26b5a59d45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr" Jul 13 14:28:31.489751 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.489731 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e453e581-c08e-4391-87d1-af26b5a59d45-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5f2dr\" (UID: \"e453e581-c08e-4391-87d1-af26b5a59d45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr" Jul 13 14:28:31.501426 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.501403 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t5g7\" (UniqueName: \"kubernetes.io/projected/e453e581-c08e-4391-87d1-af26b5a59d45-kube-api-access-8t5g7\") pod \"kuadrant-console-plugin-66b5774985-5f2dr\" (UID: \"e453e581-c08e-4391-87d1-af26b5a59d45\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr" Jul 13 14:28:31.644889 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.644817 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr" Jul 13 14:28:31.779164 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:31.779136 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr"] Jul 13 14:28:31.780475 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:28:31.780451 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode453e581_c08e_4391_87d1_af26b5a59d45.slice/crio-0d14b5273ee274c985625fdb6e111f293e03456da46922e9c7b60a0aa15be253 WatchSource:0}: Error finding container 0d14b5273ee274c985625fdb6e111f293e03456da46922e9c7b60a0aa15be253: Status 404 returned error can't find the container with id 0d14b5273ee274c985625fdb6e111f293e03456da46922e9c7b60a0aa15be253 Jul 13 14:28:32.667451 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:32.667421 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-t5p2z" Jul 13 14:28:32.718179 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:32.718143 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr" event={"ID":"e453e581-c08e-4391-87d1-af26b5a59d45","Type":"ContainerStarted","Data":"0d14b5273ee274c985625fdb6e111f293e03456da46922e9c7b60a0aa15be253"} Jul 13 14:28:33.130677 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:33.130624 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:33.225814 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:33.225757 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:39.701454 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:39.701424 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:39.768903 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:39.768866 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:39.876286 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:39.876221 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:40.925046 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:40.925011 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v"] Jul 13 14:28:40.928645 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:40.928619 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:40.941278 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:40.941218 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v"] Jul 13 14:28:40.974489 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:40.974463 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k9hx\" (UniqueName: \"kubernetes.io/projected/9b9f73ad-ab77-4ac8-aa29-32fce139f356-kube-api-access-8k9hx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6ph8v\" (UID: \"9b9f73ad-ab77-4ac8-aa29-32fce139f356\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:40.974686 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:40.974503 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b9f73ad-ab77-4ac8-aa29-32fce139f356-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6ph8v\" (UID: \"9b9f73ad-ab77-4ac8-aa29-32fce139f356\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:41.075768 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.075731 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k9hx\" (UniqueName: \"kubernetes.io/projected/9b9f73ad-ab77-4ac8-aa29-32fce139f356-kube-api-access-8k9hx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6ph8v\" (UID: \"9b9f73ad-ab77-4ac8-aa29-32fce139f356\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:41.075949 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.075810 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b9f73ad-ab77-4ac8-aa29-32fce139f356-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6ph8v\" (UID: \"9b9f73ad-ab77-4ac8-aa29-32fce139f356\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:41.076162 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.076143 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b9f73ad-ab77-4ac8-aa29-32fce139f356-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6ph8v\" (UID: \"9b9f73ad-ab77-4ac8-aa29-32fce139f356\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:41.085966 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.085941 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k9hx\" (UniqueName: \"kubernetes.io/projected/9b9f73ad-ab77-4ac8-aa29-32fce139f356-kube-api-access-8k9hx\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-6ph8v\" (UID: \"9b9f73ad-ab77-4ac8-aa29-32fce139f356\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:41.243384 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.243294 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:41.768075 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.768002 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v"] Jul 13 14:28:41.771072 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.771037 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v"] Jul 13 14:28:41.790395 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.790369 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p"] Jul 13 14:28:41.790644 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.790593 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" podUID="cad8c529-921a-4491-863b-4ba4336cf869" containerName="manager" containerID="cri-o://727654c690ab2e05a49c3415ab406205f899f57b56bb282f8e0b21fa786f34e8" gracePeriod=2 Jul 13 14:28:41.793658 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:41.793624 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-txh6p\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="cad8c529-921a-4491-863b-4ba4336cf869" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:41.797171 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.797113 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p"] Jul 13 14:28:41.804382 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.804361 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd"] Jul 13 14:28:41.804826 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.804808 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cad8c529-921a-4491-863b-4ba4336cf869" containerName="manager" Jul 13 14:28:41.804826 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.804826 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cad8c529-921a-4491-863b-4ba4336cf869" containerName="manager" Jul 13 14:28:41.804987 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.804903 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cad8c529-921a-4491-863b-4ba4336cf869" containerName="manager" Jul 13 14:28:41.810612 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.810585 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" Jul 13 14:28:41.814319 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:41.814283 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-txh6p\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="cad8c529-921a-4491-863b-4ba4336cf869" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:41.817063 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.817035 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx"] Jul 13 14:28:41.818028 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.817245 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" podUID="91c28b56-8058-45ba-b1b0-1d12036565ae" containerName="manager" containerID="cri-o://6cff8f9922f9424987048d2c83a3e52d1a75fcb41e273240292823a16f963794" gracePeriod=2 Jul 13 14:28:41.826525 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.826489 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx"] Jul 13 14:28:41.829755 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.829731 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd"] Jul 13 14:28:41.834089 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.833962 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4"] Jul 13 14:28:41.835541 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.834411 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91c28b56-8058-45ba-b1b0-1d12036565ae" containerName="manager" Jul 13 14:28:41.835541 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.834431 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="91c28b56-8058-45ba-b1b0-1d12036565ae" containerName="manager" Jul 13 14:28:41.835541 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.834527 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="91c28b56-8058-45ba-b1b0-1d12036565ae" containerName="manager" Jul 13 14:28:41.838110 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.838083 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" Jul 13 14:28:41.848412 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:41.848380 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-txh6p\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="cad8c529-921a-4491-863b-4ba4336cf869" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:41.865438 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.865370 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4"] Jul 13 14:28:41.878312 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:41.878279 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-smhqx\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="91c28b56-8058-45ba-b1b0-1d12036565ae" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" Jul 13 14:28:41.883388 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.883357 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8hrc\" (UniqueName: \"kubernetes.io/projected/eb008a83-21da-4fff-a3be-ba549ace6011-kube-api-access-k8hrc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f5gd4\" (UID: \"eb008a83-21da-4fff-a3be-ba549ace6011\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" Jul 13 14:28:41.883504 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.883432 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb008a83-21da-4fff-a3be-ba549ace6011-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f5gd4\" (UID: \"eb008a83-21da-4fff-a3be-ba549ace6011\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" Jul 13 14:28:41.883572 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.883500 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78845fa0-c3e2-403b-aa71-17d63a5534b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-95ddd\" (UID: \"78845fa0-c3e2-403b-aa71-17d63a5534b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" Jul 13 14:28:41.883630 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.883574 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgxv5\" (UniqueName: \"kubernetes.io/projected/78845fa0-c3e2-403b-aa71-17d63a5534b8-kube-api-access-qgxv5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-95ddd\" (UID: \"78845fa0-c3e2-403b-aa71-17d63a5534b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" Jul 13 14:28:41.985045 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.985002 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgxv5\" (UniqueName: \"kubernetes.io/projected/78845fa0-c3e2-403b-aa71-17d63a5534b8-kube-api-access-qgxv5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-95ddd\" (UID: \"78845fa0-c3e2-403b-aa71-17d63a5534b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" Jul 13 14:28:41.985460 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.985081 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8hrc\" (UniqueName: \"kubernetes.io/projected/eb008a83-21da-4fff-a3be-ba549ace6011-kube-api-access-k8hrc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f5gd4\" (UID: \"eb008a83-21da-4fff-a3be-ba549ace6011\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" Jul 13 14:28:41.985460 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.985136 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb008a83-21da-4fff-a3be-ba549ace6011-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f5gd4\" (UID: \"eb008a83-21da-4fff-a3be-ba549ace6011\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" Jul 13 14:28:41.985460 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.985183 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78845fa0-c3e2-403b-aa71-17d63a5534b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-95ddd\" (UID: \"78845fa0-c3e2-403b-aa71-17d63a5534b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" Jul 13 14:28:41.985624 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.985578 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb008a83-21da-4fff-a3be-ba549ace6011-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f5gd4\" (UID: \"eb008a83-21da-4fff-a3be-ba549ace6011\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" Jul 13 14:28:41.985675 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.985616 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78845fa0-c3e2-403b-aa71-17d63a5534b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-95ddd\" (UID: \"78845fa0-c3e2-403b-aa71-17d63a5534b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" Jul 13 14:28:41.997192 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.997163 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgxv5\" (UniqueName: \"kubernetes.io/projected/78845fa0-c3e2-403b-aa71-17d63a5534b8-kube-api-access-qgxv5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-95ddd\" (UID: \"78845fa0-c3e2-403b-aa71-17d63a5534b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" Jul 13 14:28:41.997347 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:41.997241 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8hrc\" (UniqueName: \"kubernetes.io/projected/eb008a83-21da-4fff-a3be-ba549ace6011-kube-api-access-k8hrc\") pod \"kuadrant-operator-controller-manager-8684c9ff49-f5gd4\" (UID: \"eb008a83-21da-4fff-a3be-ba549ace6011\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" Jul 13 14:28:42.206590 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:42.206552 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" Jul 13 14:28:42.214261 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:42.214241 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" Jul 13 14:28:43.586525 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:43.586493 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:43.690006 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:43.689972 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:43.791199 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:43.791169 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:43.888344 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:43.888257 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:46.507051 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:46.507009 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:47.570153 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:47.570111 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:47.671278 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:47.671241 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:52.204536 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:52.204483 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:52.305611 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:52.305577 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:28:55.110160 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.109874 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" Jul 13 14:28:55.112634 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:55.112592 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-smhqx\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="91c28b56-8058-45ba-b1b0-1d12036565ae" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" Jul 13 14:28:55.113554 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.113530 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:55.116268 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:55.116237 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-smhqx\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="91c28b56-8058-45ba-b1b0-1d12036565ae" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" Jul 13 14:28:55.118476 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:55.118452 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-txh6p\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="cad8c529-921a-4491-863b-4ba4336cf869" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:55.148681 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.148655 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd"] Jul 13 14:28:55.149515 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:28:55.149493 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78845fa0_c3e2_403b_aa71_17d63a5534b8.slice/crio-007d3616fa15900f686649bf57fa0c9ac003dad48b7edc50238d866dcdd7b9cb WatchSource:0}: Error finding container 007d3616fa15900f686649bf57fa0c9ac003dad48b7edc50238d866dcdd7b9cb: Status 404 returned error can't find the container with id 007d3616fa15900f686649bf57fa0c9ac003dad48b7edc50238d866dcdd7b9cb Jul 13 14:28:55.163318 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.163291 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4"] Jul 13 14:28:55.164780 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:28:55.164758 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb008a83_21da_4fff_a3be_ba549ace6011.slice/crio-23ecb06fbfe8d877aa329dc5e61ea8687c58177e631b7f06cadcaf91e6e8e8d5 WatchSource:0}: Error finding container 23ecb06fbfe8d877aa329dc5e61ea8687c58177e631b7f06cadcaf91e6e8e8d5: Status 404 returned error can't find the container with id 23ecb06fbfe8d877aa329dc5e61ea8687c58177e631b7f06cadcaf91e6e8e8d5 Jul 13 14:28:55.203905 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.203878 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cad8c529-921a-4491-863b-4ba4336cf869-extensions-socket-volume\") pod \"cad8c529-921a-4491-863b-4ba4336cf869\" (UID: \"cad8c529-921a-4491-863b-4ba4336cf869\") " Jul 13 14:28:55.204002 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.203941 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz2nf\" (UniqueName: \"kubernetes.io/projected/91c28b56-8058-45ba-b1b0-1d12036565ae-kube-api-access-zz2nf\") pod \"91c28b56-8058-45ba-b1b0-1d12036565ae\" (UID: \"91c28b56-8058-45ba-b1b0-1d12036565ae\") " Jul 13 14:28:55.204002 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.203978 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmdrf\" (UniqueName: \"kubernetes.io/projected/cad8c529-921a-4491-863b-4ba4336cf869-kube-api-access-qmdrf\") pod \"cad8c529-921a-4491-863b-4ba4336cf869\" (UID: \"cad8c529-921a-4491-863b-4ba4336cf869\") " Jul 13 14:28:55.204349 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.204255 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad8c529-921a-4491-863b-4ba4336cf869-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cad8c529-921a-4491-863b-4ba4336cf869" (UID: "cad8c529-921a-4491-863b-4ba4336cf869"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:28:55.206003 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.205979 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c28b56-8058-45ba-b1b0-1d12036565ae-kube-api-access-zz2nf" (OuterVolumeSpecName: "kube-api-access-zz2nf") pod "91c28b56-8058-45ba-b1b0-1d12036565ae" (UID: "91c28b56-8058-45ba-b1b0-1d12036565ae"). InnerVolumeSpecName "kube-api-access-zz2nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:28:55.206143 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.206121 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad8c529-921a-4491-863b-4ba4336cf869-kube-api-access-qmdrf" (OuterVolumeSpecName: "kube-api-access-qmdrf") pod "cad8c529-921a-4491-863b-4ba4336cf869" (UID: "cad8c529-921a-4491-863b-4ba4336cf869"). InnerVolumeSpecName "kube-api-access-qmdrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:28:55.305335 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.305306 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cad8c529-921a-4491-863b-4ba4336cf869-extensions-socket-volume\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:28:55.305335 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.305332 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz2nf\" (UniqueName: \"kubernetes.io/projected/91c28b56-8058-45ba-b1b0-1d12036565ae-kube-api-access-zz2nf\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:28:55.305335 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.305341 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmdrf\" (UniqueName: \"kubernetes.io/projected/cad8c529-921a-4491-863b-4ba4336cf869-kube-api-access-qmdrf\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:28:55.849122 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.849086 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" event={"ID":"78845fa0-c3e2-403b-aa71-17d63a5534b8","Type":"ContainerStarted","Data":"f28efcf02cd6519c9e6ad3032bdd7f23f321bece38041dc54f49c44338e65e0a"} Jul 13 14:28:55.849323 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.849153 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" Jul 13 14:28:55.849323 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.849178 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" event={"ID":"78845fa0-c3e2-403b-aa71-17d63a5534b8","Type":"ContainerStarted","Data":"007d3616fa15900f686649bf57fa0c9ac003dad48b7edc50238d866dcdd7b9cb"} Jul 13 14:28:55.850364 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.850342 2566 generic.go:356] "Generic (PLEG): container finished" podID="91c28b56-8058-45ba-b1b0-1d12036565ae" containerID="6cff8f9922f9424987048d2c83a3e52d1a75fcb41e273240292823a16f963794" exitCode=0 Jul 13 14:28:55.850428 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.850376 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" Jul 13 14:28:55.850428 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.850417 2566 scope.go:117] "RemoveContainer" containerID="6cff8f9922f9424987048d2c83a3e52d1a75fcb41e273240292823a16f963794" Jul 13 14:28:55.852476 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.852315 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr" event={"ID":"e453e581-c08e-4391-87d1-af26b5a59d45","Type":"ContainerStarted","Data":"d8733886fc70858aea97e6959d4ce538b9dcd867d4ea7e31036e6bf93285e2ab"} Jul 13 14:28:55.854248 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.854221 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" event={"ID":"eb008a83-21da-4fff-a3be-ba549ace6011","Type":"ContainerStarted","Data":"6a131ecb0ad2ce8af64c008f6ae7e808392547ba51744348c9861719794b0fcf"} Jul 13 14:28:55.854370 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.854274 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" event={"ID":"eb008a83-21da-4fff-a3be-ba549ace6011","Type":"ContainerStarted","Data":"23ecb06fbfe8d877aa329dc5e61ea8687c58177e631b7f06cadcaf91e6e8e8d5"} Jul 13 14:28:55.854370 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.854323 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" Jul 13 14:28:55.855537 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.855515 2566 generic.go:356] "Generic (PLEG): container finished" podID="cad8c529-921a-4491-863b-4ba4336cf869" containerID="727654c690ab2e05a49c3415ab406205f899f57b56bb282f8e0b21fa786f34e8" exitCode=0 Jul 13 14:28:55.855626 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.855569 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:55.862109 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.862012 2566 scope.go:117] "RemoveContainer" containerID="6cff8f9922f9424987048d2c83a3e52d1a75fcb41e273240292823a16f963794" Jul 13 14:28:55.863911 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:55.863884 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cff8f9922f9424987048d2c83a3e52d1a75fcb41e273240292823a16f963794\": container with ID starting with 6cff8f9922f9424987048d2c83a3e52d1a75fcb41e273240292823a16f963794 not found: ID does not exist" containerID="6cff8f9922f9424987048d2c83a3e52d1a75fcb41e273240292823a16f963794" Jul 13 14:28:55.864007 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.863919 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cff8f9922f9424987048d2c83a3e52d1a75fcb41e273240292823a16f963794"} err="failed to get container status \"6cff8f9922f9424987048d2c83a3e52d1a75fcb41e273240292823a16f963794\": rpc error: code = NotFound desc = could not find container \"6cff8f9922f9424987048d2c83a3e52d1a75fcb41e273240292823a16f963794\": container with ID starting with 6cff8f9922f9424987048d2c83a3e52d1a75fcb41e273240292823a16f963794 not found: ID does not exist" Jul 13 14:28:55.864007 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.863943 2566 scope.go:117] "RemoveContainer" containerID="727654c690ab2e05a49c3415ab406205f899f57b56bb282f8e0b21fa786f34e8" Jul 13 14:28:55.871776 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.871759 2566 scope.go:117] "RemoveContainer" containerID="727654c690ab2e05a49c3415ab406205f899f57b56bb282f8e0b21fa786f34e8" Jul 13 14:28:55.872034 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:55.872015 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727654c690ab2e05a49c3415ab406205f899f57b56bb282f8e0b21fa786f34e8\": container with ID starting with 727654c690ab2e05a49c3415ab406205f899f57b56bb282f8e0b21fa786f34e8 not found: ID does not exist" containerID="727654c690ab2e05a49c3415ab406205f899f57b56bb282f8e0b21fa786f34e8" Jul 13 14:28:55.872084 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.872043 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727654c690ab2e05a49c3415ab406205f899f57b56bb282f8e0b21fa786f34e8"} err="failed to get container status \"727654c690ab2e05a49c3415ab406205f899f57b56bb282f8e0b21fa786f34e8\": rpc error: code = NotFound desc = could not find container \"727654c690ab2e05a49c3415ab406205f899f57b56bb282f8e0b21fa786f34e8\": container with ID starting with 727654c690ab2e05a49c3415ab406205f899f57b56bb282f8e0b21fa786f34e8 not found: ID does not exist" Jul 13 14:28:55.884204 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.884162 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" podStartSLOduration=14.884149135 podStartE2EDuration="14.884149135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:28:55.880506078 +0000 UTC m=+477.564926380" watchObservedRunningTime="2026-07-13 14:28:55.884149135 +0000 UTC m=+477.568569441" Jul 13 14:28:55.884425 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:55.884401 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-smhqx\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="91c28b56-8058-45ba-b1b0-1d12036565ae" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" Jul 13 14:28:55.888350 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:55.888325 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-txh6p\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="cad8c529-921a-4491-863b-4ba4336cf869" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:55.892720 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:55.892697 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-txh6p\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="cad8c529-921a-4491-863b-4ba4336cf869" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-txh6p" Jul 13 14:28:55.916464 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.916425 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5f2dr" podStartSLOduration=1.6761872009999998 podStartE2EDuration="24.91641124s" totalImagesPullingTime="23.240224039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:28:55.914449066 +0000 UTC m=+477.598869371" watchObservedRunningTime="2026-07-13 14:28:55.91641124 +0000 UTC m=+477.600831547" Jul 13 14:28:55.942073 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:55.942003 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" podStartSLOduration=14.941983803 podStartE2EDuration="14.941983803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:28:55.940835985 +0000 UTC m=+477.625256308" watchObservedRunningTime="2026-07-13 14:28:55.941983803 +0000 UTC m=+477.626404110" Jul 13 14:28:55.944083 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:55.944050 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-smhqx\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="91c28b56-8058-45ba-b1b0-1d12036565ae" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-smhqx" Jul 13 14:28:56.866480 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:56.866451 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91c28b56-8058-45ba-b1b0-1d12036565ae" path="/var/lib/kubelet/pods/91c28b56-8058-45ba-b1b0-1d12036565ae/volumes" Jul 13 14:28:56.866855 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:56.866839 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cad8c529-921a-4491-863b-4ba4336cf869" path="/var/lib/kubelet/pods/cad8c529-921a-4491-863b-4ba4336cf869/volumes" Jul 13 14:28:57.520811 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:57.520753 2566 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-6ph8v_kuadrant-system_9b9f73ad-ab77-4ac8-aa29-32fce139f356_0(8984f0d24f24658ed2d9f07c9f947ddc5905519a2a3b70ee8871778f97229703): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-6ph8v to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 13 14:28:57.520913 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:57.520831 2566 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-6ph8v_kuadrant-system_9b9f73ad-ab77-4ac8-aa29-32fce139f356_0(8984f0d24f24658ed2d9f07c9f947ddc5905519a2a3b70ee8871778f97229703): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-6ph8v to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:57.867389 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:57.867360 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:57.872350 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:57.872322 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6ph8v\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="9b9f73ad-ab77-4ac8-aa29-32fce139f356" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:57.875391 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:57.875371 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:57.880296 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:57.880273 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6ph8v\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="9b9f73ad-ab77-4ac8-aa29-32fce139f356" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:58.027865 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:58.027826 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k9hx\" (UniqueName: \"kubernetes.io/projected/9b9f73ad-ab77-4ac8-aa29-32fce139f356-kube-api-access-8k9hx\") pod \"9b9f73ad-ab77-4ac8-aa29-32fce139f356\" (UID: \"9b9f73ad-ab77-4ac8-aa29-32fce139f356\") " Jul 13 14:28:58.027865 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:58.027867 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b9f73ad-ab77-4ac8-aa29-32fce139f356-extensions-socket-volume\") pod \"9b9f73ad-ab77-4ac8-aa29-32fce139f356\" (UID: \"9b9f73ad-ab77-4ac8-aa29-32fce139f356\") " Jul 13 14:28:58.028177 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:58.028142 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9f73ad-ab77-4ac8-aa29-32fce139f356-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9b9f73ad-ab77-4ac8-aa29-32fce139f356" (UID: "9b9f73ad-ab77-4ac8-aa29-32fce139f356"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:28:58.029745 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:58.029721 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9f73ad-ab77-4ac8-aa29-32fce139f356-kube-api-access-8k9hx" (OuterVolumeSpecName: "kube-api-access-8k9hx") pod "9b9f73ad-ab77-4ac8-aa29-32fce139f356" (UID: "9b9f73ad-ab77-4ac8-aa29-32fce139f356"). InnerVolumeSpecName "kube-api-access-8k9hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:28:58.128613 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:58.128545 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k9hx\" (UniqueName: \"kubernetes.io/projected/9b9f73ad-ab77-4ac8-aa29-32fce139f356-kube-api-access-8k9hx\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:28:58.128613 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:58.128569 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b9f73ad-ab77-4ac8-aa29-32fce139f356-extensions-socket-volume\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:28:58.867956 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:58.867920 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b9f73ad-ab77-4ac8-aa29-32fce139f356" path="/var/lib/kubelet/pods/9b9f73ad-ab77-4ac8-aa29-32fce139f356/volumes" Jul 13 14:28:58.869332 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:58.869303 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6ph8v\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="9b9f73ad-ab77-4ac8-aa29-32fce139f356" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:58.872309 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:28:58.872288 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:58.876685 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:58.876660 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6ph8v\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="9b9f73ad-ab77-4ac8-aa29-32fce139f356" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:28:58.878759 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:28:58.878733 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-6ph8v\" is forbidden: User \"system:node:ip-10-0-131-188.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-188.ec2.internal' and this object" podUID="9b9f73ad-ab77-4ac8-aa29-32fce139f356" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-6ph8v" Jul 13 14:29:06.867222 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:06.867197 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" Jul 13 14:29:06.867593 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:06.867254 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" Jul 13 14:29:06.959689 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:06.959655 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4"] Jul 13 14:29:06.960071 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:06.960036 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" podUID="eb008a83-21da-4fff-a3be-ba549ace6011" containerName="manager" containerID="cri-o://6a131ecb0ad2ce8af64c008f6ae7e808392547ba51744348c9861719794b0fcf" gracePeriod=10 Jul 13 14:29:07.203274 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.203252 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" Jul 13 14:29:07.264981 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.264929 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld"] Jul 13 14:29:07.265947 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.265919 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb008a83-21da-4fff-a3be-ba549ace6011" containerName="manager" Jul 13 14:29:07.265947 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.265948 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb008a83-21da-4fff-a3be-ba549ace6011" containerName="manager" Jul 13 14:29:07.266296 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.266276 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb008a83-21da-4fff-a3be-ba549ace6011" containerName="manager" Jul 13 14:29:07.269601 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.269576 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" Jul 13 14:29:07.278553 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.278529 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld"] Jul 13 14:29:07.303477 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.303458 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8hrc\" (UniqueName: \"kubernetes.io/projected/eb008a83-21da-4fff-a3be-ba549ace6011-kube-api-access-k8hrc\") pod \"eb008a83-21da-4fff-a3be-ba549ace6011\" (UID: \"eb008a83-21da-4fff-a3be-ba549ace6011\") " Jul 13 14:29:07.303599 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.303497 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb008a83-21da-4fff-a3be-ba549ace6011-extensions-socket-volume\") pod \"eb008a83-21da-4fff-a3be-ba549ace6011\" (UID: \"eb008a83-21da-4fff-a3be-ba549ace6011\") " Jul 13 14:29:07.303906 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.303885 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb008a83-21da-4fff-a3be-ba549ace6011-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "eb008a83-21da-4fff-a3be-ba549ace6011" (UID: "eb008a83-21da-4fff-a3be-ba549ace6011"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:29:07.305380 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.305355 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb008a83-21da-4fff-a3be-ba549ace6011-kube-api-access-k8hrc" (OuterVolumeSpecName: "kube-api-access-k8hrc") pod "eb008a83-21da-4fff-a3be-ba549ace6011" (UID: "eb008a83-21da-4fff-a3be-ba549ace6011"). InnerVolumeSpecName "kube-api-access-k8hrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:29:07.404415 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.404342 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d3b35e1-204a-492f-b894-0ef94c7f8c1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-m8hld\" (UID: \"4d3b35e1-204a-492f-b894-0ef94c7f8c1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" Jul 13 14:29:07.404415 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.404374 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndfx6\" (UniqueName: \"kubernetes.io/projected/4d3b35e1-204a-492f-b894-0ef94c7f8c1e-kube-api-access-ndfx6\") pod \"kuadrant-operator-controller-manager-86878c769f-m8hld\" (UID: \"4d3b35e1-204a-492f-b894-0ef94c7f8c1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" Jul 13 14:29:07.404581 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.404440 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8hrc\" (UniqueName: \"kubernetes.io/projected/eb008a83-21da-4fff-a3be-ba549ace6011-kube-api-access-k8hrc\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:29:07.404581 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.404453 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb008a83-21da-4fff-a3be-ba549ace6011-extensions-socket-volume\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:29:07.505276 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.505249 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndfx6\" (UniqueName: \"kubernetes.io/projected/4d3b35e1-204a-492f-b894-0ef94c7f8c1e-kube-api-access-ndfx6\") pod \"kuadrant-operator-controller-manager-86878c769f-m8hld\" (UID: \"4d3b35e1-204a-492f-b894-0ef94c7f8c1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" Jul 13 14:29:07.505393 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.505369 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d3b35e1-204a-492f-b894-0ef94c7f8c1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-m8hld\" (UID: \"4d3b35e1-204a-492f-b894-0ef94c7f8c1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" Jul 13 14:29:07.505731 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.505713 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d3b35e1-204a-492f-b894-0ef94c7f8c1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-m8hld\" (UID: \"4d3b35e1-204a-492f-b894-0ef94c7f8c1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" Jul 13 14:29:07.517315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.517290 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndfx6\" (UniqueName: \"kubernetes.io/projected/4d3b35e1-204a-492f-b894-0ef94c7f8c1e-kube-api-access-ndfx6\") pod \"kuadrant-operator-controller-manager-86878c769f-m8hld\" (UID: \"4d3b35e1-204a-492f-b894-0ef94c7f8c1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" Jul 13 14:29:07.580499 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.580475 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" Jul 13 14:29:07.738344 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:29:07.738304 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3b35e1_204a_492f_b894_0ef94c7f8c1e.slice/crio-c791a6ef3c1be02bc20f6f48e3b6381a8e262b31adf6465b38922f7d4c57febb WatchSource:0}: Error finding container c791a6ef3c1be02bc20f6f48e3b6381a8e262b31adf6465b38922f7d4c57febb: Status 404 returned error can't find the container with id c791a6ef3c1be02bc20f6f48e3b6381a8e262b31adf6465b38922f7d4c57febb Jul 13 14:29:07.739877 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.739852 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld"] Jul 13 14:29:07.916683 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.916589 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" event={"ID":"4d3b35e1-204a-492f-b894-0ef94c7f8c1e","Type":"ContainerStarted","Data":"09352639405ef8794a46ab6e80528481469383dda5c7f710f7ebda69981acf0a"} Jul 13 14:29:07.916683 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.916645 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" event={"ID":"4d3b35e1-204a-492f-b894-0ef94c7f8c1e","Type":"ContainerStarted","Data":"c791a6ef3c1be02bc20f6f48e3b6381a8e262b31adf6465b38922f7d4c57febb"} Jul 13 14:29:07.917125 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.916693 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" Jul 13 14:29:07.917977 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.917955 2566 generic.go:356] "Generic (PLEG): container finished" podID="eb008a83-21da-4fff-a3be-ba549ace6011" containerID="6a131ecb0ad2ce8af64c008f6ae7e808392547ba51744348c9861719794b0fcf" exitCode=0 Jul 13 14:29:07.918098 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.918024 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" Jul 13 14:29:07.918098 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.918038 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" event={"ID":"eb008a83-21da-4fff-a3be-ba549ace6011","Type":"ContainerDied","Data":"6a131ecb0ad2ce8af64c008f6ae7e808392547ba51744348c9861719794b0fcf"} Jul 13 14:29:07.918098 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.918080 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4" event={"ID":"eb008a83-21da-4fff-a3be-ba549ace6011","Type":"ContainerDied","Data":"23ecb06fbfe8d877aa329dc5e61ea8687c58177e631b7f06cadcaf91e6e8e8d5"} Jul 13 14:29:07.918245 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.918101 2566 scope.go:117] "RemoveContainer" containerID="6a131ecb0ad2ce8af64c008f6ae7e808392547ba51744348c9861719794b0fcf" Jul 13 14:29:07.926634 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.926615 2566 scope.go:117] "RemoveContainer" containerID="6a131ecb0ad2ce8af64c008f6ae7e808392547ba51744348c9861719794b0fcf" Jul 13 14:29:07.926890 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:29:07.926873 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a131ecb0ad2ce8af64c008f6ae7e808392547ba51744348c9861719794b0fcf\": container with ID starting with 6a131ecb0ad2ce8af64c008f6ae7e808392547ba51744348c9861719794b0fcf not found: ID does not exist" containerID="6a131ecb0ad2ce8af64c008f6ae7e808392547ba51744348c9861719794b0fcf" Jul 13 14:29:07.926951 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:07.926896 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a131ecb0ad2ce8af64c008f6ae7e808392547ba51744348c9861719794b0fcf"} err="failed to get container status \"6a131ecb0ad2ce8af64c008f6ae7e808392547ba51744348c9861719794b0fcf\": rpc error: code = NotFound desc = could not find container \"6a131ecb0ad2ce8af64c008f6ae7e808392547ba51744348c9861719794b0fcf\": container with ID starting with 6a131ecb0ad2ce8af64c008f6ae7e808392547ba51744348c9861719794b0fcf not found: ID does not exist" Jul 13 14:29:08.013484 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:08.013134 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" podStartSLOduration=1.013112585 podStartE2EDuration="1.013112585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:29:07.990311605 +0000 UTC m=+489.674731911" watchObservedRunningTime="2026-07-13 14:29:08.013112585 +0000 UTC m=+489.697532905" Jul 13 14:29:08.015603 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:08.015580 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4"] Jul 13 14:29:08.017530 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:08.017507 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-f5gd4"] Jul 13 14:29:08.867580 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:08.867540 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb008a83-21da-4fff-a3be-ba549ace6011" path="/var/lib/kubelet/pods/eb008a83-21da-4fff-a3be-ba549ace6011/volumes" Jul 13 14:29:18.925927 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:18.925889 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" Jul 13 14:29:18.997720 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:18.997690 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd"] Jul 13 14:29:18.997968 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:18.997944 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" podUID="78845fa0-c3e2-403b-aa71-17d63a5534b8" containerName="manager" containerID="cri-o://f28efcf02cd6519c9e6ad3032bdd7f23f321bece38041dc54f49c44338e65e0a" gracePeriod=10 Jul 13 14:29:19.788259 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:19.788232 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" Jul 13 14:29:19.796833 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:19.796811 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgxv5\" (UniqueName: \"kubernetes.io/projected/78845fa0-c3e2-403b-aa71-17d63a5534b8-kube-api-access-qgxv5\") pod \"78845fa0-c3e2-403b-aa71-17d63a5534b8\" (UID: \"78845fa0-c3e2-403b-aa71-17d63a5534b8\") " Jul 13 14:29:19.796948 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:19.796856 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78845fa0-c3e2-403b-aa71-17d63a5534b8-extensions-socket-volume\") pod \"78845fa0-c3e2-403b-aa71-17d63a5534b8\" (UID: \"78845fa0-c3e2-403b-aa71-17d63a5534b8\") " Jul 13 14:29:19.797227 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:19.797205 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78845fa0-c3e2-403b-aa71-17d63a5534b8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "78845fa0-c3e2-403b-aa71-17d63a5534b8" (UID: "78845fa0-c3e2-403b-aa71-17d63a5534b8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:29:19.798753 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:19.798736 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78845fa0-c3e2-403b-aa71-17d63a5534b8-kube-api-access-qgxv5" (OuterVolumeSpecName: "kube-api-access-qgxv5") pod "78845fa0-c3e2-403b-aa71-17d63a5534b8" (UID: "78845fa0-c3e2-403b-aa71-17d63a5534b8"). InnerVolumeSpecName "kube-api-access-qgxv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:29:19.897951 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:19.897921 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/78845fa0-c3e2-403b-aa71-17d63a5534b8-extensions-socket-volume\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:29:19.897951 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:19.897947 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgxv5\" (UniqueName: \"kubernetes.io/projected/78845fa0-c3e2-403b-aa71-17d63a5534b8-kube-api-access-qgxv5\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:29:19.984463 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:19.984393 2566 generic.go:356] "Generic (PLEG): container finished" podID="78845fa0-c3e2-403b-aa71-17d63a5534b8" containerID="f28efcf02cd6519c9e6ad3032bdd7f23f321bece38041dc54f49c44338e65e0a" exitCode=0 Jul 13 14:29:19.984463 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:19.984439 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" Jul 13 14:29:19.984907 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:19.984475 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" event={"ID":"78845fa0-c3e2-403b-aa71-17d63a5534b8","Type":"ContainerDied","Data":"f28efcf02cd6519c9e6ad3032bdd7f23f321bece38041dc54f49c44338e65e0a"} Jul 13 14:29:19.984907 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:19.984530 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd" event={"ID":"78845fa0-c3e2-403b-aa71-17d63a5534b8","Type":"ContainerDied","Data":"007d3616fa15900f686649bf57fa0c9ac003dad48b7edc50238d866dcdd7b9cb"} Jul 13 14:29:19.984907 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:19.984554 2566 scope.go:117] "RemoveContainer" containerID="f28efcf02cd6519c9e6ad3032bdd7f23f321bece38041dc54f49c44338e65e0a" Jul 13 14:29:19.993775 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:19.993754 2566 scope.go:117] "RemoveContainer" containerID="f28efcf02cd6519c9e6ad3032bdd7f23f321bece38041dc54f49c44338e65e0a" Jul 13 14:29:19.994120 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:29:19.994102 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28efcf02cd6519c9e6ad3032bdd7f23f321bece38041dc54f49c44338e65e0a\": container with ID starting with f28efcf02cd6519c9e6ad3032bdd7f23f321bece38041dc54f49c44338e65e0a not found: ID does not exist" containerID="f28efcf02cd6519c9e6ad3032bdd7f23f321bece38041dc54f49c44338e65e0a" Jul 13 14:29:19.994179 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:19.994128 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28efcf02cd6519c9e6ad3032bdd7f23f321bece38041dc54f49c44338e65e0a"} err="failed to get container status \"f28efcf02cd6519c9e6ad3032bdd7f23f321bece38041dc54f49c44338e65e0a\": rpc error: code = NotFound desc = could not find container \"f28efcf02cd6519c9e6ad3032bdd7f23f321bece38041dc54f49c44338e65e0a\": container with ID starting with f28efcf02cd6519c9e6ad3032bdd7f23f321bece38041dc54f49c44338e65e0a not found: ID does not exist" Jul 13 14:29:20.018859 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:20.018827 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd"] Jul 13 14:29:20.025205 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:20.025182 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-95ddd"] Jul 13 14:29:20.867223 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:20.867188 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78845fa0-c3e2-403b-aa71-17d63a5534b8" path="/var/lib/kubelet/pods/78845fa0-c3e2-403b-aa71-17d63a5534b8/volumes" Jul 13 14:29:23.571049 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.571021 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv"] Jul 13 14:29:23.571426 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.571364 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78845fa0-c3e2-403b-aa71-17d63a5534b8" containerName="manager" Jul 13 14:29:23.571426 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.571376 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="78845fa0-c3e2-403b-aa71-17d63a5534b8" containerName="manager" Jul 13 14:29:23.571500 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.571445 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="78845fa0-c3e2-403b-aa71-17d63a5534b8" containerName="manager" Jul 13 14:29:23.573493 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.573473 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.580223 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.580204 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rgvzj\"" Jul 13 14:29:23.589575 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.589550 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv"] Jul 13 14:29:23.620683 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.620657 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58626a7f-6711-430f-8a37-ea4262636ae7-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.620840 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.620693 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58626a7f-6711-430f-8a37-ea4262636ae7-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.620840 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.620721 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58626a7f-6711-430f-8a37-ea4262636ae7-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.620840 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.620803 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58626a7f-6711-430f-8a37-ea4262636ae7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.620840 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.620832 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mkwq\" (UniqueName: \"kubernetes.io/projected/58626a7f-6711-430f-8a37-ea4262636ae7-kube-api-access-5mkwq\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.621008 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.620852 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58626a7f-6711-430f-8a37-ea4262636ae7-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.621008 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.620868 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58626a7f-6711-430f-8a37-ea4262636ae7-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.621008 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.620894 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58626a7f-6711-430f-8a37-ea4262636ae7-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.621008 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.620909 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58626a7f-6711-430f-8a37-ea4262636ae7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.721227 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.721194 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58626a7f-6711-430f-8a37-ea4262636ae7-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.721227 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.721230 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58626a7f-6711-430f-8a37-ea4262636ae7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.721451 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.721252 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58626a7f-6711-430f-8a37-ea4262636ae7-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.721451 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.721297 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58626a7f-6711-430f-8a37-ea4262636ae7-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.721451 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.721347 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58626a7f-6711-430f-8a37-ea4262636ae7-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.721451 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.721402 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58626a7f-6711-430f-8a37-ea4262636ae7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.721451 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.721428 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mkwq\" (UniqueName: \"kubernetes.io/projected/58626a7f-6711-430f-8a37-ea4262636ae7-kube-api-access-5mkwq\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.721694 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.721459 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58626a7f-6711-430f-8a37-ea4262636ae7-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.721694 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.721488 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58626a7f-6711-430f-8a37-ea4262636ae7-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.721694 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.721663 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58626a7f-6711-430f-8a37-ea4262636ae7-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.721881 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.721735 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58626a7f-6711-430f-8a37-ea4262636ae7-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.722007 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.721980 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58626a7f-6711-430f-8a37-ea4262636ae7-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.722007 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.721994 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58626a7f-6711-430f-8a37-ea4262636ae7-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.722299 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.722278 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58626a7f-6711-430f-8a37-ea4262636ae7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.723562 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.723540 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58626a7f-6711-430f-8a37-ea4262636ae7-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.723856 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.723837 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58626a7f-6711-430f-8a37-ea4262636ae7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.734653 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.734626 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mkwq\" (UniqueName: \"kubernetes.io/projected/58626a7f-6711-430f-8a37-ea4262636ae7-kube-api-access-5mkwq\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.734894 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.734872 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58626a7f-6711-430f-8a37-ea4262636ae7-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-nsxgv\" (UID: \"58626a7f-6711-430f-8a37-ea4262636ae7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.792635 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.792597 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:29:23.883946 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.883862 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:23.893253 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:23.893207 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:29:24.025075 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:24.025046 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv"] Jul 13 14:29:24.026396 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:29:24.026374 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58626a7f_6711_430f_8a37_ea4262636ae7.slice/crio-489aff4e19366ae63528cfc48dabaf87b6c2248716406e7127a7e9c38e6ef854 WatchSource:0}: Error finding container 489aff4e19366ae63528cfc48dabaf87b6c2248716406e7127a7e9c38e6ef854: Status 404 returned error can't find the container with id 489aff4e19366ae63528cfc48dabaf87b6c2248716406e7127a7e9c38e6ef854 Jul 13 14:29:24.028221 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:24.028193 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:29:24.028316 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:24.028247 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:29:24.028316 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:24.028277 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:29:25.014384 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:25.014343 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" event={"ID":"58626a7f-6711-430f-8a37-ea4262636ae7","Type":"ContainerStarted","Data":"bfd78246423f34f03bcbe373a0e7e0eb9302febcbead3d4560e127025cb8432d"} Jul 13 14:29:25.014807 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:25.014398 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" event={"ID":"58626a7f-6711-430f-8a37-ea4262636ae7","Type":"ContainerStarted","Data":"489aff4e19366ae63528cfc48dabaf87b6c2248716406e7127a7e9c38e6ef854"} Jul 13 14:29:25.041062 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:25.041016 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" podStartSLOduration=2.041000941 podStartE2EDuration="2.041000941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:29:25.038866316 +0000 UTC m=+506.723286621" watchObservedRunningTime="2026-07-13 14:29:25.041000941 +0000 UTC m=+506.725421248" Jul 13 14:29:25.884726 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:25.884689 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:25.889379 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:25.889351 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:26.019108 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:26.019071 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:26.020075 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:26.020053 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:29:35.453015 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:35.452984 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:29:37.838994 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:37.838960 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:29:37.852948 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:37.852921 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fdmcr"] Jul 13 14:29:37.934428 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:37.934397 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fdmcr"] Jul 13 14:29:37.934683 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:37.934525 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" Jul 13 14:29:37.937259 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:37.937232 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 13 14:29:37.943691 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:37.943586 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fdmcr"] Jul 13 14:29:37.947559 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:37.947535 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:29:38.013943 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:38.013868 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:29:38.040029 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:38.040004 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/523839ec-bf78-4094-b516-ef5425e9fb3d-config-file\") pod \"limitador-limitador-64c94fd64f-fdmcr\" (UID: \"523839ec-bf78-4094-b516-ef5425e9fb3d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" Jul 13 14:29:38.040196 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:38.040082 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcbfl\" (UniqueName: \"kubernetes.io/projected/523839ec-bf78-4094-b516-ef5425e9fb3d-kube-api-access-bcbfl\") pod \"limitador-limitador-64c94fd64f-fdmcr\" (UID: \"523839ec-bf78-4094-b516-ef5425e9fb3d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" Jul 13 14:29:38.140738 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:38.140708 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcbfl\" (UniqueName: \"kubernetes.io/projected/523839ec-bf78-4094-b516-ef5425e9fb3d-kube-api-access-bcbfl\") pod \"limitador-limitador-64c94fd64f-fdmcr\" (UID: \"523839ec-bf78-4094-b516-ef5425e9fb3d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" Jul 13 14:29:38.140928 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:38.140764 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/523839ec-bf78-4094-b516-ef5425e9fb3d-config-file\") pod \"limitador-limitador-64c94fd64f-fdmcr\" (UID: \"523839ec-bf78-4094-b516-ef5425e9fb3d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" Jul 13 14:29:38.141460 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:38.141442 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/523839ec-bf78-4094-b516-ef5425e9fb3d-config-file\") pod \"limitador-limitador-64c94fd64f-fdmcr\" (UID: \"523839ec-bf78-4094-b516-ef5425e9fb3d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" Jul 13 14:29:38.151158 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:38.151129 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcbfl\" (UniqueName: \"kubernetes.io/projected/523839ec-bf78-4094-b516-ef5425e9fb3d-kube-api-access-bcbfl\") pod \"limitador-limitador-64c94fd64f-fdmcr\" (UID: \"523839ec-bf78-4094-b516-ef5425e9fb3d\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" Jul 13 14:29:38.247646 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:38.247612 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" Jul 13 14:29:38.396067 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:38.396037 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fdmcr"] Jul 13 14:29:38.397092 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:29:38.397070 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523839ec_bf78_4094_b516_ef5425e9fb3d.slice/crio-3a860fdaffd2d9f16175dc9f8dc18a226508f4f3ce0f4ce487098a16092a9bce WatchSource:0}: Error finding container 3a860fdaffd2d9f16175dc9f8dc18a226508f4f3ce0f4ce487098a16092a9bce: Status 404 returned error can't find the container with id 3a860fdaffd2d9f16175dc9f8dc18a226508f4f3ce0f4ce487098a16092a9bce Jul 13 14:29:38.766872 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:38.766840 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:29:38.869077 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:38.869049 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:29:38.963937 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:38.963905 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-nzgpw"] Jul 13 14:29:38.968429 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:38.968413 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nzgpw" Jul 13 14:29:38.971291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:38.971269 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7k45z\"" Jul 13 14:29:38.976588 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:38.976569 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nzgpw"] Jul 13 14:29:39.047904 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:39.047831 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sxsr\" (UniqueName: \"kubernetes.io/projected/bf4ec629-72b1-4fde-9f3f-617466597baf-kube-api-access-2sxsr\") pod \"authorino-75d556bd77-nzgpw\" (UID: \"bf4ec629-72b1-4fde-9f3f-617466597baf\") " pod="kuadrant-system/authorino-75d556bd77-nzgpw" Jul 13 14:29:39.083201 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:39.083173 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" event={"ID":"523839ec-bf78-4094-b516-ef5425e9fb3d","Type":"ContainerStarted","Data":"3a860fdaffd2d9f16175dc9f8dc18a226508f4f3ce0f4ce487098a16092a9bce"} Jul 13 14:29:39.148616 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:39.148587 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sxsr\" (UniqueName: \"kubernetes.io/projected/bf4ec629-72b1-4fde-9f3f-617466597baf-kube-api-access-2sxsr\") pod \"authorino-75d556bd77-nzgpw\" (UID: \"bf4ec629-72b1-4fde-9f3f-617466597baf\") " pod="kuadrant-system/authorino-75d556bd77-nzgpw" Jul 13 14:29:39.158537 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:39.158512 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sxsr\" (UniqueName: \"kubernetes.io/projected/bf4ec629-72b1-4fde-9f3f-617466597baf-kube-api-access-2sxsr\") pod \"authorino-75d556bd77-nzgpw\" (UID: \"bf4ec629-72b1-4fde-9f3f-617466597baf\") " pod="kuadrant-system/authorino-75d556bd77-nzgpw" Jul 13 14:29:39.278031 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:39.277999 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nzgpw" Jul 13 14:29:39.453676 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:39.453647 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nzgpw"] Jul 13 14:29:40.090217 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:40.090177 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nzgpw" event={"ID":"bf4ec629-72b1-4fde-9f3f-617466597baf","Type":"ContainerStarted","Data":"b82df084a1b227489a59c23d6b2cf7f041b4d767677b0b32d9a7662da35ef7d2"} Jul 13 14:29:42.106344 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:42.106288 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" event={"ID":"523839ec-bf78-4094-b516-ef5425e9fb3d","Type":"ContainerStarted","Data":"488900b713933105ec44e3c4ec962bd387cf081a13b4df0ebb6b97d49deec3a9"} Jul 13 14:29:42.106777 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:42.106469 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" Jul 13 14:29:42.126314 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:42.126265 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" podStartSLOduration=2.095356343 podStartE2EDuration="5.126246691s" totalImagesPullingTime="3.030890348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:29:42.124608458 +0000 UTC m=+523.809028794" watchObservedRunningTime="2026-07-13 14:29:42.126246691 +0000 UTC m=+523.810666998" Jul 13 14:29:44.118149 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:44.118117 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nzgpw" event={"ID":"bf4ec629-72b1-4fde-9f3f-617466597baf","Type":"ContainerStarted","Data":"f0ed9e40636f2896f9a535fc5568168f425b1c7b2d1c295328e177fbfb0b5dbd"} Jul 13 14:29:44.151940 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:44.151891 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-nzgpw" podStartSLOduration=2.304315641 podStartE2EDuration="6.151877146s" totalImagesPullingTime="3.847561505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:29:44.150534541 +0000 UTC m=+525.834954829" watchObservedRunningTime="2026-07-13 14:29:44.151877146 +0000 UTC m=+525.836297452" Jul 13 14:29:53.113101 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:53.113031 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" Jul 13 14:29:55.724353 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:55.724310 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fdmcr"] Jul 13 14:29:55.724882 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:55.724513 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" podUID="523839ec-bf78-4094-b516-ef5425e9fb3d" containerName="limitador" containerID="cri-o://488900b713933105ec44e3c4ec962bd387cf081a13b4df0ebb6b97d49deec3a9" gracePeriod=30 Jul 13 14:29:56.178639 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:56.178601 2566 generic.go:356] "Generic (PLEG): container finished" podID="523839ec-bf78-4094-b516-ef5425e9fb3d" containerID="488900b713933105ec44e3c4ec962bd387cf081a13b4df0ebb6b97d49deec3a9" exitCode=0 Jul 13 14:29:56.178806 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:56.178674 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" event={"ID":"523839ec-bf78-4094-b516-ef5425e9fb3d","Type":"ContainerDied","Data":"488900b713933105ec44e3c4ec962bd387cf081a13b4df0ebb6b97d49deec3a9"} Jul 13 14:29:56.271826 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:56.271780 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" Jul 13 14:29:56.402904 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:56.402866 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcbfl\" (UniqueName: \"kubernetes.io/projected/523839ec-bf78-4094-b516-ef5425e9fb3d-kube-api-access-bcbfl\") pod \"523839ec-bf78-4094-b516-ef5425e9fb3d\" (UID: \"523839ec-bf78-4094-b516-ef5425e9fb3d\") " Jul 13 14:29:56.403183 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:56.402968 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/523839ec-bf78-4094-b516-ef5425e9fb3d-config-file\") pod \"523839ec-bf78-4094-b516-ef5425e9fb3d\" (UID: \"523839ec-bf78-4094-b516-ef5425e9fb3d\") " Jul 13 14:29:56.403625 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:56.403594 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/523839ec-bf78-4094-b516-ef5425e9fb3d-config-file" (OuterVolumeSpecName: "config-file") pod "523839ec-bf78-4094-b516-ef5425e9fb3d" (UID: "523839ec-bf78-4094-b516-ef5425e9fb3d"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:29:56.406038 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:56.406012 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523839ec-bf78-4094-b516-ef5425e9fb3d-kube-api-access-bcbfl" (OuterVolumeSpecName: "kube-api-access-bcbfl") pod "523839ec-bf78-4094-b516-ef5425e9fb3d" (UID: "523839ec-bf78-4094-b516-ef5425e9fb3d"). InnerVolumeSpecName "kube-api-access-bcbfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:29:56.503965 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:56.503931 2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/523839ec-bf78-4094-b516-ef5425e9fb3d-config-file\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:29:56.503965 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:56.503963 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcbfl\" (UniqueName: \"kubernetes.io/projected/523839ec-bf78-4094-b516-ef5425e9fb3d-kube-api-access-bcbfl\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:29:57.185402 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:57.185314 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" event={"ID":"523839ec-bf78-4094-b516-ef5425e9fb3d","Type":"ContainerDied","Data":"3a860fdaffd2d9f16175dc9f8dc18a226508f4f3ce0f4ce487098a16092a9bce"} Jul 13 14:29:57.185402 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:57.185370 2566 scope.go:117] "RemoveContainer" containerID="488900b713933105ec44e3c4ec962bd387cf081a13b4df0ebb6b97d49deec3a9" Jul 13 14:29:57.185402 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:57.185321 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-fdmcr" Jul 13 14:29:57.205666 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:57.205620 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fdmcr"] Jul 13 14:29:57.211869 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:57.211839 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-fdmcr"] Jul 13 14:29:57.954876 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:57.954845 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:29:58.047580 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:58.047543 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:29:58.562583 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:58.562552 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:29:58.866916 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:29:58.866889 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="523839ec-bf78-4094-b516-ef5425e9fb3d" path="/var/lib/kubelet/pods/523839ec-bf78-4094-b516-ef5425e9fb3d/volumes" Jul 13 14:30:15.942129 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:15.942096 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:30:16.231853 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:16.231746 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gdj8j"] Jul 13 14:30:16.232130 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:16.232116 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="523839ec-bf78-4094-b516-ef5425e9fb3d" containerName="limitador" Jul 13 14:30:16.232183 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:16.232132 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="523839ec-bf78-4094-b516-ef5425e9fb3d" containerName="limitador" Jul 13 14:30:16.232219 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:16.232201 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="523839ec-bf78-4094-b516-ef5425e9fb3d" containerName="limitador" Jul 13 14:30:16.260173 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:16.260144 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:30:16.303241 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:16.303210 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gdj8j"] Jul 13 14:30:16.303396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:16.303297 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gdj8j" Jul 13 14:30:16.367811 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:16.367767 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c67f\" (UniqueName: \"kubernetes.io/projected/ba8002d6-7a96-49c2-82ce-2920f2589941-kube-api-access-9c67f\") pod \"authorino-5d4b67c446-gdj8j\" (UID: \"ba8002d6-7a96-49c2-82ce-2920f2589941\") " pod="kuadrant-system/authorino-5d4b67c446-gdj8j" Jul 13 14:30:16.469074 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:16.469042 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c67f\" (UniqueName: \"kubernetes.io/projected/ba8002d6-7a96-49c2-82ce-2920f2589941-kube-api-access-9c67f\") pod \"authorino-5d4b67c446-gdj8j\" (UID: \"ba8002d6-7a96-49c2-82ce-2920f2589941\") " pod="kuadrant-system/authorino-5d4b67c446-gdj8j" Jul 13 14:30:16.478735 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:16.478706 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c67f\" (UniqueName: \"kubernetes.io/projected/ba8002d6-7a96-49c2-82ce-2920f2589941-kube-api-access-9c67f\") pod \"authorino-5d4b67c446-gdj8j\" (UID: \"ba8002d6-7a96-49c2-82ce-2920f2589941\") " pod="kuadrant-system/authorino-5d4b67c446-gdj8j" Jul 13 14:30:16.529478 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:16.529402 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gdj8j"] Jul 13 14:30:16.529650 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:16.529638 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gdj8j" Jul 13 14:30:16.665174 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:16.665149 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gdj8j"] Jul 13 14:30:16.667388 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:30:16.667361 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba8002d6_7a96_49c2_82ce_2920f2589941.slice/crio-aeca1cf2af77447bd0f6f258a977afc9ed787ea26b916187123cb0bf289b0747 WatchSource:0}: Error finding container aeca1cf2af77447bd0f6f258a977afc9ed787ea26b916187123cb0bf289b0747: Status 404 returned error can't find the container with id aeca1cf2af77447bd0f6f258a977afc9ed787ea26b916187123cb0bf289b0747 Jul 13 14:30:17.284516 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:17.284479 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gdj8j" event={"ID":"ba8002d6-7a96-49c2-82ce-2920f2589941","Type":"ContainerStarted","Data":"142a551068d093925bceb33a0079d83685b7797a7be1bd1eb70289d432db58a4"} Jul 13 14:30:17.284516 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:17.284524 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gdj8j" event={"ID":"ba8002d6-7a96-49c2-82ce-2920f2589941","Type":"ContainerStarted","Data":"aeca1cf2af77447bd0f6f258a977afc9ed787ea26b916187123cb0bf289b0747"} Jul 13 14:30:17.285018 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:17.284542 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-gdj8j" podUID="ba8002d6-7a96-49c2-82ce-2920f2589941" containerName="authorino" containerID="cri-o://142a551068d093925bceb33a0079d83685b7797a7be1bd1eb70289d432db58a4" gracePeriod=30 Jul 13 14:30:17.302368 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:17.302328 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-gdj8j" podStartSLOduration=0.86119901 podStartE2EDuration="1.302312364s" totalImagesPullingTime="441.113354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:30:17.300164663 +0000 UTC m=+558.984584992" watchObservedRunningTime="2026-07-13 14:30:17.302312364 +0000 UTC m=+558.986732718" Jul 13 14:30:17.522542 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:17.522516 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gdj8j" Jul 13 14:30:17.579867 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:17.579838 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c67f\" (UniqueName: \"kubernetes.io/projected/ba8002d6-7a96-49c2-82ce-2920f2589941-kube-api-access-9c67f\") pod \"ba8002d6-7a96-49c2-82ce-2920f2589941\" (UID: \"ba8002d6-7a96-49c2-82ce-2920f2589941\") " Jul 13 14:30:17.581892 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:17.581865 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba8002d6-7a96-49c2-82ce-2920f2589941-kube-api-access-9c67f" (OuterVolumeSpecName: "kube-api-access-9c67f") pod "ba8002d6-7a96-49c2-82ce-2920f2589941" (UID: "ba8002d6-7a96-49c2-82ce-2920f2589941"). InnerVolumeSpecName "kube-api-access-9c67f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:30:17.681365 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:17.681300 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c67f\" (UniqueName: \"kubernetes.io/projected/ba8002d6-7a96-49c2-82ce-2920f2589941-kube-api-access-9c67f\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:30:17.867627 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:17.867595 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nzgpw"] Jul 13 14:30:17.867884 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:17.867857 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-nzgpw" podUID="bf4ec629-72b1-4fde-9f3f-617466597baf" containerName="authorino" containerID="cri-o://f0ed9e40636f2896f9a535fc5568168f425b1c7b2d1c295328e177fbfb0b5dbd" gracePeriod=30 Jul 13 14:30:18.115418 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.115396 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nzgpw" Jul 13 14:30:18.184954 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.184924 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sxsr\" (UniqueName: \"kubernetes.io/projected/bf4ec629-72b1-4fde-9f3f-617466597baf-kube-api-access-2sxsr\") pod \"bf4ec629-72b1-4fde-9f3f-617466597baf\" (UID: \"bf4ec629-72b1-4fde-9f3f-617466597baf\") " Jul 13 14:30:18.186942 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.186916 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf4ec629-72b1-4fde-9f3f-617466597baf-kube-api-access-2sxsr" (OuterVolumeSpecName: "kube-api-access-2sxsr") pod "bf4ec629-72b1-4fde-9f3f-617466597baf" (UID: "bf4ec629-72b1-4fde-9f3f-617466597baf"). InnerVolumeSpecName "kube-api-access-2sxsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:30:18.285525 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.285455 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sxsr\" (UniqueName: \"kubernetes.io/projected/bf4ec629-72b1-4fde-9f3f-617466597baf-kube-api-access-2sxsr\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:30:18.290549 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.290511 2566 generic.go:356] "Generic (PLEG): container finished" podID="bf4ec629-72b1-4fde-9f3f-617466597baf" containerID="f0ed9e40636f2896f9a535fc5568168f425b1c7b2d1c295328e177fbfb0b5dbd" exitCode=0 Jul 13 14:30:18.290697 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.290554 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-nzgpw" Jul 13 14:30:18.290697 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.290596 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nzgpw" event={"ID":"bf4ec629-72b1-4fde-9f3f-617466597baf","Type":"ContainerDied","Data":"f0ed9e40636f2896f9a535fc5568168f425b1c7b2d1c295328e177fbfb0b5dbd"} Jul 13 14:30:18.290697 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.290635 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-nzgpw" event={"ID":"bf4ec629-72b1-4fde-9f3f-617466597baf","Type":"ContainerDied","Data":"b82df084a1b227489a59c23d6b2cf7f041b4d767677b0b32d9a7662da35ef7d2"} Jul 13 14:30:18.290697 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.290651 2566 scope.go:117] "RemoveContainer" containerID="f0ed9e40636f2896f9a535fc5568168f425b1c7b2d1c295328e177fbfb0b5dbd" Jul 13 14:30:18.292168 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.292143 2566 generic.go:356] "Generic (PLEG): container finished" podID="ba8002d6-7a96-49c2-82ce-2920f2589941" containerID="142a551068d093925bceb33a0079d83685b7797a7be1bd1eb70289d432db58a4" exitCode=0 Jul 13 14:30:18.292282 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.292207 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-gdj8j" Jul 13 14:30:18.292282 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.292223 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gdj8j" event={"ID":"ba8002d6-7a96-49c2-82ce-2920f2589941","Type":"ContainerDied","Data":"142a551068d093925bceb33a0079d83685b7797a7be1bd1eb70289d432db58a4"} Jul 13 14:30:18.292282 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.292251 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-gdj8j" event={"ID":"ba8002d6-7a96-49c2-82ce-2920f2589941","Type":"ContainerDied","Data":"aeca1cf2af77447bd0f6f258a977afc9ed787ea26b916187123cb0bf289b0747"} Jul 13 14:30:18.300291 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.300272 2566 scope.go:117] "RemoveContainer" containerID="f0ed9e40636f2896f9a535fc5568168f425b1c7b2d1c295328e177fbfb0b5dbd" Jul 13 14:30:18.300560 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:30:18.300537 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ed9e40636f2896f9a535fc5568168f425b1c7b2d1c295328e177fbfb0b5dbd\": container with ID starting with f0ed9e40636f2896f9a535fc5568168f425b1c7b2d1c295328e177fbfb0b5dbd not found: ID does not exist" containerID="f0ed9e40636f2896f9a535fc5568168f425b1c7b2d1c295328e177fbfb0b5dbd" Jul 13 14:30:18.300734 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.300567 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ed9e40636f2896f9a535fc5568168f425b1c7b2d1c295328e177fbfb0b5dbd"} err="failed to get container status \"f0ed9e40636f2896f9a535fc5568168f425b1c7b2d1c295328e177fbfb0b5dbd\": rpc error: code = NotFound desc = could not find container \"f0ed9e40636f2896f9a535fc5568168f425b1c7b2d1c295328e177fbfb0b5dbd\": container with ID starting with f0ed9e40636f2896f9a535fc5568168f425b1c7b2d1c295328e177fbfb0b5dbd not found: ID does not exist" Jul 13 14:30:18.300734 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.300591 2566 scope.go:117] "RemoveContainer" containerID="142a551068d093925bceb33a0079d83685b7797a7be1bd1eb70289d432db58a4" Jul 13 14:30:18.308607 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.308591 2566 scope.go:117] "RemoveContainer" containerID="142a551068d093925bceb33a0079d83685b7797a7be1bd1eb70289d432db58a4" Jul 13 14:30:18.308861 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:30:18.308844 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"142a551068d093925bceb33a0079d83685b7797a7be1bd1eb70289d432db58a4\": container with ID starting with 142a551068d093925bceb33a0079d83685b7797a7be1bd1eb70289d432db58a4 not found: ID does not exist" containerID="142a551068d093925bceb33a0079d83685b7797a7be1bd1eb70289d432db58a4" Jul 13 14:30:18.308914 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.308868 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"142a551068d093925bceb33a0079d83685b7797a7be1bd1eb70289d432db58a4"} err="failed to get container status \"142a551068d093925bceb33a0079d83685b7797a7be1bd1eb70289d432db58a4\": rpc error: code = NotFound desc = could not find container \"142a551068d093925bceb33a0079d83685b7797a7be1bd1eb70289d432db58a4\": container with ID starting with 142a551068d093925bceb33a0079d83685b7797a7be1bd1eb70289d432db58a4 not found: ID does not exist" Jul 13 14:30:18.317338 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.317317 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nzgpw"] Jul 13 14:30:18.325515 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.325493 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-nzgpw"] Jul 13 14:30:18.339914 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.339894 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gdj8j"] Jul 13 14:30:18.344390 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.344372 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-gdj8j"] Jul 13 14:30:18.867675 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.867644 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba8002d6-7a96-49c2-82ce-2920f2589941" path="/var/lib/kubelet/pods/ba8002d6-7a96-49c2-82ce-2920f2589941/volumes" Jul 13 14:30:18.867969 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:18.867955 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf4ec629-72b1-4fde-9f3f-617466597baf" path="/var/lib/kubelet/pods/bf4ec629-72b1-4fde-9f3f-617466597baf/volumes" Jul 13 14:30:23.107685 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.107651 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:30:23.139450 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.139420 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f586fdfd-9qnkp"] Jul 13 14:30:23.139834 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.139817 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba8002d6-7a96-49c2-82ce-2920f2589941" containerName="authorino" Jul 13 14:30:23.139834 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.139833 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba8002d6-7a96-49c2-82ce-2920f2589941" containerName="authorino" Jul 13 14:30:23.139966 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.139847 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf4ec629-72b1-4fde-9f3f-617466597baf" containerName="authorino" Jul 13 14:30:23.139966 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.139853 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf4ec629-72b1-4fde-9f3f-617466597baf" containerName="authorino" Jul 13 14:30:23.139966 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.139905 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf4ec629-72b1-4fde-9f3f-617466597baf" containerName="authorino" Jul 13 14:30:23.139966 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.139915 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba8002d6-7a96-49c2-82ce-2920f2589941" containerName="authorino" Jul 13 14:30:23.142003 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.141987 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f586fdfd-9qnkp" Jul 13 14:30:23.144680 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.144662 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 13 14:30:23.144904 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.144886 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nlvgx\"" Jul 13 14:30:23.145137 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.145123 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 13 14:30:23.156313 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.156292 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f586fdfd-9qnkp"] Jul 13 14:30:23.210021 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.209992 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:30:23.228242 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.228219 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkkvd\" (UniqueName: \"kubernetes.io/projected/f2334ea5-5bdc-42fe-be41-324fa90a59f5-kube-api-access-xkkvd\") pod \"maas-controller-7f586fdfd-9qnkp\" (UID: \"f2334ea5-5bdc-42fe-be41-324fa90a59f5\") " pod="opendatahub/maas-controller-7f586fdfd-9qnkp" Jul 13 14:30:23.228350 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.228254 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2334ea5-5bdc-42fe-be41-324fa90a59f5-webhook-cert\") pod \"maas-controller-7f586fdfd-9qnkp\" (UID: \"f2334ea5-5bdc-42fe-be41-324fa90a59f5\") " pod="opendatahub/maas-controller-7f586fdfd-9qnkp" Jul 13 14:30:23.307034 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.307003 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:30:23.329026 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.328993 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2334ea5-5bdc-42fe-be41-324fa90a59f5-webhook-cert\") pod \"maas-controller-7f586fdfd-9qnkp\" (UID: \"f2334ea5-5bdc-42fe-be41-324fa90a59f5\") " pod="opendatahub/maas-controller-7f586fdfd-9qnkp" Jul 13 14:30:23.329172 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:30:23.329155 2566 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 13 14:30:23.329242 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.329176 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkkvd\" (UniqueName: \"kubernetes.io/projected/f2334ea5-5bdc-42fe-be41-324fa90a59f5-kube-api-access-xkkvd\") pod \"maas-controller-7f586fdfd-9qnkp\" (UID: \"f2334ea5-5bdc-42fe-be41-324fa90a59f5\") " pod="opendatahub/maas-controller-7f586fdfd-9qnkp" Jul 13 14:30:23.329242 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:30:23.329213 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2334ea5-5bdc-42fe-be41-324fa90a59f5-webhook-cert podName:f2334ea5-5bdc-42fe-be41-324fa90a59f5 nodeName:}" failed. No retries permitted until 2026-07-13 14:30:23.829194852 +0000 UTC m=+565.513615139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/f2334ea5-5bdc-42fe-be41-324fa90a59f5-webhook-cert") pod "maas-controller-7f586fdfd-9qnkp" (UID: "f2334ea5-5bdc-42fe-be41-324fa90a59f5") : secret "maas-controller-webhook-cert" not found Jul 13 14:30:23.350610 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.350579 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkkvd\" (UniqueName: \"kubernetes.io/projected/f2334ea5-5bdc-42fe-be41-324fa90a59f5-kube-api-access-xkkvd\") pod \"maas-controller-7f586fdfd-9qnkp\" (UID: \"f2334ea5-5bdc-42fe-be41-324fa90a59f5\") " pod="opendatahub/maas-controller-7f586fdfd-9qnkp" Jul 13 14:30:23.362828 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.362753 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-786f5ff9c9-bwtzz"] Jul 13 14:30:23.364959 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.364946 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" Jul 13 14:30:23.397839 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.397817 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-786f5ff9c9-bwtzz"] Jul 13 14:30:23.429604 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.429577 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35b97602-8c8b-4a92-90e2-e6e489f41db9-webhook-cert\") pod \"maas-controller-786f5ff9c9-bwtzz\" (UID: \"35b97602-8c8b-4a92-90e2-e6e489f41db9\") " pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" Jul 13 14:30:23.429718 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.429647 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdgbm\" (UniqueName: \"kubernetes.io/projected/35b97602-8c8b-4a92-90e2-e6e489f41db9-kube-api-access-hdgbm\") pod \"maas-controller-786f5ff9c9-bwtzz\" (UID: \"35b97602-8c8b-4a92-90e2-e6e489f41db9\") " pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" Jul 13 14:30:23.530987 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.530958 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdgbm\" (UniqueName: \"kubernetes.io/projected/35b97602-8c8b-4a92-90e2-e6e489f41db9-kube-api-access-hdgbm\") pod \"maas-controller-786f5ff9c9-bwtzz\" (UID: \"35b97602-8c8b-4a92-90e2-e6e489f41db9\") " pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" Jul 13 14:30:23.531157 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.531032 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35b97602-8c8b-4a92-90e2-e6e489f41db9-webhook-cert\") pod \"maas-controller-786f5ff9c9-bwtzz\" (UID: \"35b97602-8c8b-4a92-90e2-e6e489f41db9\") " pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" Jul 13 14:30:23.533696 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.533674 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35b97602-8c8b-4a92-90e2-e6e489f41db9-webhook-cert\") pod \"maas-controller-786f5ff9c9-bwtzz\" (UID: \"35b97602-8c8b-4a92-90e2-e6e489f41db9\") " pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" Jul 13 14:30:23.542605 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.542582 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdgbm\" (UniqueName: \"kubernetes.io/projected/35b97602-8c8b-4a92-90e2-e6e489f41db9-kube-api-access-hdgbm\") pod \"maas-controller-786f5ff9c9-bwtzz\" (UID: \"35b97602-8c8b-4a92-90e2-e6e489f41db9\") " pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" Jul 13 14:30:23.583695 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.583638 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:30:23.674366 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.674282 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" Jul 13 14:30:23.806125 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.806102 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-786f5ff9c9-bwtzz"] Jul 13 14:30:23.807775 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:30:23.807753 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b97602_8c8b_4a92_90e2_e6e489f41db9.slice/crio-47585188bd93d95873e8b8821d996ed20024b84804316ecefe0376923b0c8e2e WatchSource:0}: Error finding container 47585188bd93d95873e8b8821d996ed20024b84804316ecefe0376923b0c8e2e: Status 404 returned error can't find the container with id 47585188bd93d95873e8b8821d996ed20024b84804316ecefe0376923b0c8e2e Jul 13 14:30:23.833757 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.833729 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2334ea5-5bdc-42fe-be41-324fa90a59f5-webhook-cert\") pod \"maas-controller-7f586fdfd-9qnkp\" (UID: \"f2334ea5-5bdc-42fe-be41-324fa90a59f5\") " pod="opendatahub/maas-controller-7f586fdfd-9qnkp" Jul 13 14:30:23.835962 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:23.835946 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2334ea5-5bdc-42fe-be41-324fa90a59f5-webhook-cert\") pod \"maas-controller-7f586fdfd-9qnkp\" (UID: \"f2334ea5-5bdc-42fe-be41-324fa90a59f5\") " pod="opendatahub/maas-controller-7f586fdfd-9qnkp" Jul 13 14:30:24.052388 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:24.052300 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f586fdfd-9qnkp" Jul 13 14:30:24.184933 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:24.184854 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f586fdfd-9qnkp"] Jul 13 14:30:24.187458 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:30:24.187427 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2334ea5_5bdc_42fe_be41_324fa90a59f5.slice/crio-2d18ec07ddb8e94b181125b66081eff8933f24a16f988f5560f99fc725be9f62 WatchSource:0}: Error finding container 2d18ec07ddb8e94b181125b66081eff8933f24a16f988f5560f99fc725be9f62: Status 404 returned error can't find the container with id 2d18ec07ddb8e94b181125b66081eff8933f24a16f988f5560f99fc725be9f62 Jul 13 14:30:24.325359 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:24.325325 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" event={"ID":"35b97602-8c8b-4a92-90e2-e6e489f41db9","Type":"ContainerStarted","Data":"47585188bd93d95873e8b8821d996ed20024b84804316ecefe0376923b0c8e2e"} Jul 13 14:30:24.326484 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:24.326459 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f586fdfd-9qnkp" event={"ID":"f2334ea5-5bdc-42fe-be41-324fa90a59f5","Type":"ContainerStarted","Data":"2d18ec07ddb8e94b181125b66081eff8933f24a16f988f5560f99fc725be9f62"} Jul 13 14:30:26.339989 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:26.339903 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" event={"ID":"35b97602-8c8b-4a92-90e2-e6e489f41db9","Type":"ContainerStarted","Data":"12f2fd3bd254a276debd75d486d33ad8ae12afd7d51d776385edcc7747498ea2"} Jul 13 14:30:26.340371 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:26.340031 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" Jul 13 14:30:26.341563 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:26.341539 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f586fdfd-9qnkp" event={"ID":"f2334ea5-5bdc-42fe-be41-324fa90a59f5","Type":"ContainerStarted","Data":"30e6e80bdc3503f74206201c4b2c52bd2f8864384cf962cb20fb668f633803f4"} Jul 13 14:30:26.341680 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:26.341660 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f586fdfd-9qnkp" Jul 13 14:30:26.368121 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:26.368074 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" podStartSLOduration=1.095001627 podStartE2EDuration="3.36805854s" totalImagesPullingTime="2.273056913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:30:26.364326572 +0000 UTC m=+568.048746896" watchObservedRunningTime="2026-07-13 14:30:26.36805854 +0000 UTC m=+568.052478846" Jul 13 14:30:26.403774 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:26.403724 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f586fdfd-9qnkp" podStartSLOduration=1.49998454 podStartE2EDuration="3.403706668s" totalImagesPullingTime="1.903722128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:30:26.403466856 +0000 UTC m=+568.087887162" watchObservedRunningTime="2026-07-13 14:30:26.403706668 +0000 UTC m=+568.088126974" Jul 13 14:30:27.423802 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:27.423752 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:30:27.518928 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:27.518895 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:30:33.574050 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:33.574010 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:30:35.512654 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:35.512623 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:30:35.581664 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:35.581635 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:30:37.355632 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:37.355605 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" Jul 13 14:30:37.356073 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:37.355670 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f586fdfd-9qnkp" Jul 13 14:30:37.436475 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:37.436440 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f586fdfd-9qnkp"] Jul 13 14:30:37.436675 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:37.436648 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7f586fdfd-9qnkp" podUID="f2334ea5-5bdc-42fe-be41-324fa90a59f5" containerName="manager" containerID="cri-o://30e6e80bdc3503f74206201c4b2c52bd2f8864384cf962cb20fb668f633803f4" gracePeriod=30 Jul 13 14:30:37.691591 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:37.691567 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f586fdfd-9qnkp" Jul 13 14:30:37.764268 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:37.764239 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkkvd\" (UniqueName: \"kubernetes.io/projected/f2334ea5-5bdc-42fe-be41-324fa90a59f5-kube-api-access-xkkvd\") pod \"f2334ea5-5bdc-42fe-be41-324fa90a59f5\" (UID: \"f2334ea5-5bdc-42fe-be41-324fa90a59f5\") " Jul 13 14:30:37.764412 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:37.764291 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2334ea5-5bdc-42fe-be41-324fa90a59f5-webhook-cert\") pod \"f2334ea5-5bdc-42fe-be41-324fa90a59f5\" (UID: \"f2334ea5-5bdc-42fe-be41-324fa90a59f5\") " Jul 13 14:30:37.766434 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:37.766405 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2334ea5-5bdc-42fe-be41-324fa90a59f5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f2334ea5-5bdc-42fe-be41-324fa90a59f5" (UID: "f2334ea5-5bdc-42fe-be41-324fa90a59f5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:30:37.766494 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:37.766457 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2334ea5-5bdc-42fe-be41-324fa90a59f5-kube-api-access-xkkvd" (OuterVolumeSpecName: "kube-api-access-xkkvd") pod "f2334ea5-5bdc-42fe-be41-324fa90a59f5" (UID: "f2334ea5-5bdc-42fe-be41-324fa90a59f5"). InnerVolumeSpecName "kube-api-access-xkkvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:30:37.865661 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:37.865628 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkkvd\" (UniqueName: \"kubernetes.io/projected/f2334ea5-5bdc-42fe-be41-324fa90a59f5-kube-api-access-xkkvd\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:30:37.865661 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:37.865653 2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2334ea5-5bdc-42fe-be41-324fa90a59f5-webhook-cert\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:30:38.416981 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:38.416943 2566 generic.go:356] "Generic (PLEG): container finished" podID="f2334ea5-5bdc-42fe-be41-324fa90a59f5" containerID="30e6e80bdc3503f74206201c4b2c52bd2f8864384cf962cb20fb668f633803f4" exitCode=0 Jul 13 14:30:38.417440 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:38.417003 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f586fdfd-9qnkp" Jul 13 14:30:38.417440 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:38.417029 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f586fdfd-9qnkp" event={"ID":"f2334ea5-5bdc-42fe-be41-324fa90a59f5","Type":"ContainerDied","Data":"30e6e80bdc3503f74206201c4b2c52bd2f8864384cf962cb20fb668f633803f4"} Jul 13 14:30:38.417440 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:38.417122 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f586fdfd-9qnkp" event={"ID":"f2334ea5-5bdc-42fe-be41-324fa90a59f5","Type":"ContainerDied","Data":"2d18ec07ddb8e94b181125b66081eff8933f24a16f988f5560f99fc725be9f62"} Jul 13 14:30:38.417440 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:38.417172 2566 scope.go:117] "RemoveContainer" containerID="30e6e80bdc3503f74206201c4b2c52bd2f8864384cf962cb20fb668f633803f4" Jul 13 14:30:38.426537 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:38.426518 2566 scope.go:117] "RemoveContainer" containerID="30e6e80bdc3503f74206201c4b2c52bd2f8864384cf962cb20fb668f633803f4" Jul 13 14:30:38.426776 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:30:38.426757 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e6e80bdc3503f74206201c4b2c52bd2f8864384cf962cb20fb668f633803f4\": container with ID starting with 30e6e80bdc3503f74206201c4b2c52bd2f8864384cf962cb20fb668f633803f4 not found: ID does not exist" containerID="30e6e80bdc3503f74206201c4b2c52bd2f8864384cf962cb20fb668f633803f4" Jul 13 14:30:38.426854 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:38.426803 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e6e80bdc3503f74206201c4b2c52bd2f8864384cf962cb20fb668f633803f4"} err="failed to get container status \"30e6e80bdc3503f74206201c4b2c52bd2f8864384cf962cb20fb668f633803f4\": rpc error: code = NotFound desc = could not find container \"30e6e80bdc3503f74206201c4b2c52bd2f8864384cf962cb20fb668f633803f4\": container with ID starting with 30e6e80bdc3503f74206201c4b2c52bd2f8864384cf962cb20fb668f633803f4 not found: ID does not exist" Jul 13 14:30:38.440718 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:38.440696 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f586fdfd-9qnkp"] Jul 13 14:30:38.443550 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:38.443532 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f586fdfd-9qnkp"] Jul 13 14:30:38.867313 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:38.867285 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2334ea5-5bdc-42fe-be41-324fa90a59f5" path="/var/lib/kubelet/pods/f2334ea5-5bdc-42fe-be41-324fa90a59f5/volumes" Jul 13 14:30:41.112172 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:41.112135 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:30:58.834543 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:58.834519 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:30:59.254058 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.254019 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:30:59.299973 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.299944 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx"] Jul 13 14:30:59.300289 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.300277 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2334ea5-5bdc-42fe-be41-324fa90a59f5" containerName="manager" Jul 13 14:30:59.300331 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.300290 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2334ea5-5bdc-42fe-be41-324fa90a59f5" containerName="manager" Jul 13 14:30:59.300370 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.300338 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2334ea5-5bdc-42fe-be41-324fa90a59f5" containerName="manager" Jul 13 14:30:59.303145 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.303128 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" Jul 13 14:30:59.305869 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.305846 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 13 14:30:59.305869 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.305863 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 13 14:30:59.306002 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.305877 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 13 14:30:59.316429 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.316405 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx"] Jul 13 14:30:59.446147 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.446114 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4cv8\" (UniqueName: \"kubernetes.io/projected/640fbb25-582c-4b19-9564-7c7f365b34f4-kube-api-access-c4cv8\") pod \"maas-api-65ccc6b695-mtzwx\" (UID: \"640fbb25-582c-4b19-9564-7c7f365b34f4\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" Jul 13 14:30:59.446147 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.446147 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/640fbb25-582c-4b19-9564-7c7f365b34f4-openshift-service-ca\") pod \"maas-api-65ccc6b695-mtzwx\" (UID: \"640fbb25-582c-4b19-9564-7c7f365b34f4\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" Jul 13 14:30:59.446352 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.446248 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/640fbb25-582c-4b19-9564-7c7f365b34f4-maas-api-tls\") pod \"maas-api-65ccc6b695-mtzwx\" (UID: \"640fbb25-582c-4b19-9564-7c7f365b34f4\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" Jul 13 14:30:59.547046 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.546965 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/640fbb25-582c-4b19-9564-7c7f365b34f4-openshift-service-ca\") pod \"maas-api-65ccc6b695-mtzwx\" (UID: \"640fbb25-582c-4b19-9564-7c7f365b34f4\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" Jul 13 14:30:59.547225 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.547068 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/640fbb25-582c-4b19-9564-7c7f365b34f4-maas-api-tls\") pod \"maas-api-65ccc6b695-mtzwx\" (UID: \"640fbb25-582c-4b19-9564-7c7f365b34f4\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" Jul 13 14:30:59.547225 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.547128 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4cv8\" (UniqueName: \"kubernetes.io/projected/640fbb25-582c-4b19-9564-7c7f365b34f4-kube-api-access-c4cv8\") pod \"maas-api-65ccc6b695-mtzwx\" (UID: \"640fbb25-582c-4b19-9564-7c7f365b34f4\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" Jul 13 14:30:59.547346 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:30:59.547231 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 13 14:30:59.547346 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:30:59.547306 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/640fbb25-582c-4b19-9564-7c7f365b34f4-maas-api-tls podName:640fbb25-582c-4b19-9564-7c7f365b34f4 nodeName:}" failed. No retries permitted until 2026-07-13 14:31:00.047284001 +0000 UTC m=+601.731704301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/640fbb25-582c-4b19-9564-7c7f365b34f4-maas-api-tls") pod "maas-api-65ccc6b695-mtzwx" (UID: "640fbb25-582c-4b19-9564-7c7f365b34f4") : secret "maas-api-serving-cert" not found Jul 13 14:30:59.547675 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.547656 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/640fbb25-582c-4b19-9564-7c7f365b34f4-openshift-service-ca\") pod \"maas-api-65ccc6b695-mtzwx\" (UID: \"640fbb25-582c-4b19-9564-7c7f365b34f4\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" Jul 13 14:30:59.564161 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.564132 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4cv8\" (UniqueName: \"kubernetes.io/projected/640fbb25-582c-4b19-9564-7c7f365b34f4-kube-api-access-c4cv8\") pod \"maas-api-65ccc6b695-mtzwx\" (UID: \"640fbb25-582c-4b19-9564-7c7f365b34f4\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" Jul 13 14:30:59.722636 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:30:59.722598 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:31:00.021556 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.021527 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-ft98g"] Jul 13 14:31:00.024977 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.024961 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-ft98g" Jul 13 14:31:00.028945 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.028925 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 13 14:31:00.029119 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.028928 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-j48t2\"" Jul 13 14:31:00.031267 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.031245 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-lf8ct"] Jul 13 14:31:00.034936 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.034919 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-lf8ct" Jul 13 14:31:00.041739 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.041630 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-ft98g"] Jul 13 14:31:00.050231 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.050206 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-lf8ct"] Jul 13 14:31:00.052617 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.052596 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/640fbb25-582c-4b19-9564-7c7f365b34f4-maas-api-tls\") pod \"maas-api-65ccc6b695-mtzwx\" (UID: \"640fbb25-582c-4b19-9564-7c7f365b34f4\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" Jul 13 14:31:00.054878 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.054859 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/640fbb25-582c-4b19-9564-7c7f365b34f4-maas-api-tls\") pod \"maas-api-65ccc6b695-mtzwx\" (UID: \"640fbb25-582c-4b19-9564-7c7f365b34f4\") " pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" Jul 13 14:31:00.153392 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.153364 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b4c3bb-75c3-4b39-870a-b702580263e7-plugins-config-volume\") pod \"payload-processing-58f86c7798-lf8ct\" (UID: \"95b4c3bb-75c3-4b39-870a-b702580263e7\") " pod="openshift-ingress/payload-processing-58f86c7798-lf8ct" Jul 13 14:31:00.153578 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.153408 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p6d2\" (UniqueName: \"kubernetes.io/projected/95b4c3bb-75c3-4b39-870a-b702580263e7-kube-api-access-8p6d2\") pod \"payload-processing-58f86c7798-lf8ct\" (UID: \"95b4c3bb-75c3-4b39-870a-b702580263e7\") " pod="openshift-ingress/payload-processing-58f86c7798-lf8ct" Jul 13 14:31:00.153578 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.153439 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a789508e-4ce5-46e1-8af6-f11da38dbea7-plugins-config-volume\") pod \"payload-pre-processing-7964457696-ft98g\" (UID: \"a789508e-4ce5-46e1-8af6-f11da38dbea7\") " pod="openshift-ingress/payload-pre-processing-7964457696-ft98g" Jul 13 14:31:00.153578 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.153505 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8nwf\" (UniqueName: \"kubernetes.io/projected/a789508e-4ce5-46e1-8af6-f11da38dbea7-kube-api-access-r8nwf\") pod \"payload-pre-processing-7964457696-ft98g\" (UID: \"a789508e-4ce5-46e1-8af6-f11da38dbea7\") " pod="openshift-ingress/payload-pre-processing-7964457696-ft98g" Jul 13 14:31:00.214430 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.214399 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" Jul 13 14:31:00.254412 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.254373 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b4c3bb-75c3-4b39-870a-b702580263e7-plugins-config-volume\") pod \"payload-processing-58f86c7798-lf8ct\" (UID: \"95b4c3bb-75c3-4b39-870a-b702580263e7\") " pod="openshift-ingress/payload-processing-58f86c7798-lf8ct" Jul 13 14:31:00.254587 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.254427 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p6d2\" (UniqueName: \"kubernetes.io/projected/95b4c3bb-75c3-4b39-870a-b702580263e7-kube-api-access-8p6d2\") pod \"payload-processing-58f86c7798-lf8ct\" (UID: \"95b4c3bb-75c3-4b39-870a-b702580263e7\") " pod="openshift-ingress/payload-processing-58f86c7798-lf8ct" Jul 13 14:31:00.254587 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.254463 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a789508e-4ce5-46e1-8af6-f11da38dbea7-plugins-config-volume\") pod \"payload-pre-processing-7964457696-ft98g\" (UID: \"a789508e-4ce5-46e1-8af6-f11da38dbea7\") " pod="openshift-ingress/payload-pre-processing-7964457696-ft98g" Jul 13 14:31:00.254587 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.254504 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8nwf\" (UniqueName: \"kubernetes.io/projected/a789508e-4ce5-46e1-8af6-f11da38dbea7-kube-api-access-r8nwf\") pod \"payload-pre-processing-7964457696-ft98g\" (UID: \"a789508e-4ce5-46e1-8af6-f11da38dbea7\") " pod="openshift-ingress/payload-pre-processing-7964457696-ft98g" Jul 13 14:31:00.255280 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.255255 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b4c3bb-75c3-4b39-870a-b702580263e7-plugins-config-volume\") pod \"payload-processing-58f86c7798-lf8ct\" (UID: \"95b4c3bb-75c3-4b39-870a-b702580263e7\") " pod="openshift-ingress/payload-processing-58f86c7798-lf8ct" Jul 13 14:31:00.255401 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.255385 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a789508e-4ce5-46e1-8af6-f11da38dbea7-plugins-config-volume\") pod \"payload-pre-processing-7964457696-ft98g\" (UID: \"a789508e-4ce5-46e1-8af6-f11da38dbea7\") " pod="openshift-ingress/payload-pre-processing-7964457696-ft98g" Jul 13 14:31:00.275567 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.275470 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p6d2\" (UniqueName: \"kubernetes.io/projected/95b4c3bb-75c3-4b39-870a-b702580263e7-kube-api-access-8p6d2\") pod \"payload-processing-58f86c7798-lf8ct\" (UID: \"95b4c3bb-75c3-4b39-870a-b702580263e7\") " pod="openshift-ingress/payload-processing-58f86c7798-lf8ct" Jul 13 14:31:00.281454 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.281397 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8nwf\" (UniqueName: \"kubernetes.io/projected/a789508e-4ce5-46e1-8af6-f11da38dbea7-kube-api-access-r8nwf\") pod \"payload-pre-processing-7964457696-ft98g\" (UID: \"a789508e-4ce5-46e1-8af6-f11da38dbea7\") " pod="openshift-ingress/payload-pre-processing-7964457696-ft98g" Jul 13 14:31:00.338697 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.338355 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-ft98g" Jul 13 14:31:00.346002 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.345961 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-lf8ct" Jul 13 14:31:00.384831 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.384727 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx"] Jul 13 14:31:00.385837 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:31:00.385775 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640fbb25_582c_4b19_9564_7c7f365b34f4.slice/crio-d402d8c76340bdf8ca911f3dc260a55cb15518025ba53f1a852b035afdc7ae1f WatchSource:0}: Error finding container d402d8c76340bdf8ca911f3dc260a55cb15518025ba53f1a852b035afdc7ae1f: Status 404 returned error can't find the container with id d402d8c76340bdf8ca911f3dc260a55cb15518025ba53f1a852b035afdc7ae1f Jul 13 14:31:00.498741 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.498715 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-ft98g"] Jul 13 14:31:00.500763 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:31:00.500728 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda789508e_4ce5_46e1_8af6_f11da38dbea7.slice/crio-a3d49ade1f86d2d0f5b7d64727cd9fdce392ae7f01a81f5b66efe2d8c55ff78d WatchSource:0}: Error finding container a3d49ade1f86d2d0f5b7d64727cd9fdce392ae7f01a81f5b66efe2d8c55ff78d: Status 404 returned error can't find the container with id a3d49ade1f86d2d0f5b7d64727cd9fdce392ae7f01a81f5b66efe2d8c55ff78d Jul 13 14:31:00.516476 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.516456 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-lf8ct"] Jul 13 14:31:00.517750 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:31:00.517729 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b4c3bb_75c3_4b39_870a_b702580263e7.slice/crio-d0fe062074f108f75d6526af654b4501b04b6bf999db12d530a3713c30df91fc WatchSource:0}: Error finding container d0fe062074f108f75d6526af654b4501b04b6bf999db12d530a3713c30df91fc: Status 404 returned error can't find the container with id d0fe062074f108f75d6526af654b4501b04b6bf999db12d530a3713c30df91fc Jul 13 14:31:00.524214 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.524189 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" event={"ID":"640fbb25-582c-4b19-9564-7c7f365b34f4","Type":"ContainerStarted","Data":"d402d8c76340bdf8ca911f3dc260a55cb15518025ba53f1a852b035afdc7ae1f"} Jul 13 14:31:00.525315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.525293 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-lf8ct" event={"ID":"95b4c3bb-75c3-4b39-870a-b702580263e7","Type":"ContainerStarted","Data":"d0fe062074f108f75d6526af654b4501b04b6bf999db12d530a3713c30df91fc"} Jul 13 14:31:00.526411 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.526363 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-ft98g" event={"ID":"a789508e-4ce5-46e1-8af6-f11da38dbea7","Type":"ContainerStarted","Data":"a3d49ade1f86d2d0f5b7d64727cd9fdce392ae7f01a81f5b66efe2d8c55ff78d"} Jul 13 14:31:00.948880 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:00.948840 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:31:01.014380 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:01.014351 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:31:03.404634 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:03.404597 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:31:03.574399 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:03.574366 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-lf8ct" event={"ID":"95b4c3bb-75c3-4b39-870a-b702580263e7","Type":"ContainerStarted","Data":"357dc406aa200dd3ccaa185eaf3626d571fa215db39c302c5968afdd484fc9c7"} Jul 13 14:31:03.575008 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:03.574725 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-lf8ct" Jul 13 14:31:03.576402 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:03.576373 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-ft98g" event={"ID":"a789508e-4ce5-46e1-8af6-f11da38dbea7","Type":"ContainerStarted","Data":"f4df94a912334cdc2f33094abb33e1e7eef5811fe8203aaf0e0ea0e900188b83"} Jul 13 14:31:03.576529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:03.576518 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-ft98g" Jul 13 14:31:03.578145 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:03.578121 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" event={"ID":"640fbb25-582c-4b19-9564-7c7f365b34f4","Type":"ContainerStarted","Data":"bf8d83d3411090b5056abfc5d4a27fba405e0d5cca53d54ef911a1356738ee9b"} Jul 13 14:31:03.578266 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:03.578253 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" Jul 13 14:31:03.595864 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:03.595821 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-lf8ct" podStartSLOduration=1.099469231 podStartE2EDuration="3.595806314s" totalImagesPullingTime="2.496337083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:31:03.595396501 +0000 UTC m=+605.279816807" watchObservedRunningTime="2026-07-13 14:31:03.595806314 +0000 UTC m=+605.280226617" Jul 13 14:31:03.613879 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:03.613843 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" podStartSLOduration=1.9848936190000002 podStartE2EDuration="4.61382971s" totalImagesPullingTime="2.628936091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:31:03.612376038 +0000 UTC m=+605.296796343" watchObservedRunningTime="2026-07-13 14:31:03.61382971 +0000 UTC m=+605.298250016" Jul 13 14:31:03.643808 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:03.638358 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-ft98g" podStartSLOduration=2.122224458 podStartE2EDuration="4.638338519s" totalImagesPullingTime="2.516114061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:31:03.63607975 +0000 UTC m=+605.320500057" watchObservedRunningTime="2026-07-13 14:31:03.638338519 +0000 UTC m=+605.322758827" Jul 13 14:31:08.606608 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:08.606576 2566 generic.go:356] "Generic (PLEG): container finished" podID="58626a7f-6711-430f-8a37-ea4262636ae7" containerID="bfd78246423f34f03bcbe373a0e7e0eb9302febcbead3d4560e127025cb8432d" exitCode=0 Jul 13 14:31:08.607045 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:08.606657 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" event={"ID":"58626a7f-6711-430f-8a37-ea4262636ae7","Type":"ContainerDied","Data":"bfd78246423f34f03bcbe373a0e7e0eb9302febcbead3d4560e127025cb8432d"} Jul 13 14:31:08.607100 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:08.607052 2566 scope.go:117] "RemoveContainer" containerID="bfd78246423f34f03bcbe373a0e7e0eb9302febcbead3d4560e127025cb8432d" Jul 13 14:31:08.608113 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:08.608086 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:31:08.608193 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:08.608174 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:31:08.608280 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:08.608267 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:31:09.589751 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:09.589722 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65ccc6b695-mtzwx" Jul 13 14:31:09.614159 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:09.614127 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" event={"ID":"58626a7f-6711-430f-8a37-ea4262636ae7","Type":"ContainerStarted","Data":"3634066ebc618f58ef6bad80fd17cbc96a7ea5e145c739beb7c817d4dfb99872"} Jul 13 14:31:09.884311 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:09.884221 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:31:09.885396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:09.885373 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-nsxgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 13 14:31:09.885464 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:09.885423 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" podUID="58626a7f-6711-430f-8a37-ea4262636ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 13 14:31:10.724323 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:10.724284 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:31:10.884609 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:10.884576 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-nsxgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 13 14:31:10.884806 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:10.884628 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" podUID="58626a7f-6711-430f-8a37-ea4262636ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 13 14:31:11.019947 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:11.019857 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:31:11.884889 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:11.884852 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-nsxgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 13 14:31:11.885279 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:11.884907 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" podUID="58626a7f-6711-430f-8a37-ea4262636ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 13 14:31:12.884557 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:12.884525 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-nsxgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 13 14:31:12.884721 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:12.884574 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" podUID="58626a7f-6711-430f-8a37-ea4262636ae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 13 14:31:13.582108 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:13.582069 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:31:13.888487 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:13.888410 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:31:13.889453 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:13.889432 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-nsxgv" Jul 13 14:31:14.586119 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:14.586094 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-ft98g" Jul 13 14:31:15.591992 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:15.591961 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-lf8ct" Jul 13 14:31:23.096248 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.096176 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m"] Jul 13 14:31:23.099386 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.099365 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.103408 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.103381 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 13 14:31:23.103570 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.103438 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 13 14:31:23.103570 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.103385 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-llrxn\"" Jul 13 14:31:23.103570 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.103381 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 13 14:31:23.114639 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.114617 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m"] Jul 13 14:31:23.116029 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.116005 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:31:23.154490 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.154461 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ddd3b17-c725-4c85-8878-44f8515ef818-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.154623 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.154502 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ddd3b17-c725-4c85-8878-44f8515ef818-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.154623 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.154524 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ddd3b17-c725-4c85-8878-44f8515ef818-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.154623 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.154564 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7bj7\" (UniqueName: \"kubernetes.io/projected/6ddd3b17-c725-4c85-8878-44f8515ef818-kube-api-access-b7bj7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.154725 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.154657 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ddd3b17-c725-4c85-8878-44f8515ef818-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.154725 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.154678 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ddd3b17-c725-4c85-8878-44f8515ef818-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.255580 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.255547 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ddd3b17-c725-4c85-8878-44f8515ef818-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.255580 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.255582 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ddd3b17-c725-4c85-8878-44f8515ef818-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.255836 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.255600 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7bj7\" (UniqueName: \"kubernetes.io/projected/6ddd3b17-c725-4c85-8878-44f8515ef818-kube-api-access-b7bj7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.255836 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.255641 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ddd3b17-c725-4c85-8878-44f8515ef818-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.255836 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.255657 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ddd3b17-c725-4c85-8878-44f8515ef818-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.255836 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.255695 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ddd3b17-c725-4c85-8878-44f8515ef818-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.256065 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.256039 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ddd3b17-c725-4c85-8878-44f8515ef818-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.256109 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.256086 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ddd3b17-c725-4c85-8878-44f8515ef818-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.256150 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.256118 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ddd3b17-c725-4c85-8878-44f8515ef818-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.257940 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.257922 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ddd3b17-c725-4c85-8878-44f8515ef818-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.258244 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.258226 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ddd3b17-c725-4c85-8878-44f8515ef818-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.270084 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.270057 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7bj7\" (UniqueName: \"kubernetes.io/projected/6ddd3b17-c725-4c85-8878-44f8515ef818-kube-api-access-b7bj7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m\" (UID: \"6ddd3b17-c725-4c85-8878-44f8515ef818\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.409938 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.409853 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:23.551104 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.551067 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m"] Jul 13 14:31:23.553351 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:31:23.553280 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddd3b17_c725_4c85_8878_44f8515ef818.slice/crio-f4626c6f8950af598ddce2c608c56e27f6c479fe9f3c0486431d0447de7d8d82 WatchSource:0}: Error finding container f4626c6f8950af598ddce2c608c56e27f6c479fe9f3c0486431d0447de7d8d82: Status 404 returned error can't find the container with id f4626c6f8950af598ddce2c608c56e27f6c479fe9f3c0486431d0447de7d8d82 Jul 13 14:31:23.555239 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.555220 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 14:31:23.687175 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:23.687094 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" event={"ID":"6ddd3b17-c725-4c85-8878-44f8515ef818","Type":"ContainerStarted","Data":"f4626c6f8950af598ddce2c608c56e27f6c479fe9f3c0486431d0447de7d8d82"} Jul 13 14:31:25.700727 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:25.700691 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" event={"ID":"6ddd3b17-c725-4c85-8878-44f8515ef818","Type":"ContainerStarted","Data":"ce832c696408e32f3fae86c9b1eefe262905f49776ec8eb9c0d8456b3b44bf26"} Jul 13 14:31:25.720089 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:25.720045 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" podStartSLOduration=0.985057704 podStartE2EDuration="2.720030307s" totalImagesPullingTime="1.734972603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:31:25.719118853 +0000 UTC m=+627.403539160" watchObservedRunningTime="2026-07-13 14:31:25.720030307 +0000 UTC m=+627.404450612" Jul 13 14:31:33.410825 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:33.410762 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:33.411428 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:33.410848 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:33.423414 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:33.423388 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:33.754325 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:33.754242 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m" Jul 13 14:31:37.333423 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:37.333390 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:31:57.392691 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.392660 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz"] Jul 13 14:31:57.399556 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.399536 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.402694 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.402670 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 13 14:31:57.406092 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.406071 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz"] Jul 13 14:31:57.425897 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.425876 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:31:57.446223 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.446195 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.446337 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.446230 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwpkp\" (UniqueName: \"kubernetes.io/projected/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-kube-api-access-nwpkp\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.446337 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.446261 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.446337 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.446300 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.446337 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.446335 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.446494 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.446361 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.547197 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.547167 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.547359 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.547208 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.547405 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.547349 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.547455 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.547401 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.547511 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.547456 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.547511 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.547490 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwpkp\" (UniqueName: \"kubernetes.io/projected/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-kube-api-access-nwpkp\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.547635 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.547613 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.547747 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.547725 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.547845 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.547768 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.549572 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.549538 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.550074 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.550050 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.557296 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.557269 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwpkp\" (UniqueName: \"kubernetes.io/projected/50e9a061-8bd0-4582-92d7-9cea7b0b0ec4-kube-api-access-nwpkp\") pod \"e2e-trlp-test-simulated-kserve-598466548d-rrqrz\" (UID: \"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.710517 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.710441 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:31:57.851808 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.851759 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz"] Jul 13 14:31:57.853266 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:31:57.853238 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e9a061_8bd0_4582_92d7_9cea7b0b0ec4.slice/crio-9a50a58ffdc460779d064817c93f463a72a30afa387ebd2e534a522c5a5a021e WatchSource:0}: Error finding container 9a50a58ffdc460779d064817c93f463a72a30afa387ebd2e534a522c5a5a021e: Status 404 returned error can't find the container with id 9a50a58ffdc460779d064817c93f463a72a30afa387ebd2e534a522c5a5a021e Jul 13 14:31:57.866244 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:57.866218 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" event={"ID":"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4","Type":"ContainerStarted","Data":"9a50a58ffdc460779d064817c93f463a72a30afa387ebd2e534a522c5a5a021e"} Jul 13 14:31:58.876816 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:58.876771 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" event={"ID":"50e9a061-8bd0-4582-92d7-9cea7b0b0ec4","Type":"ContainerStarted","Data":"c36f033b1d2026b1265042a14a6d15e04225a99a19f3b6b778af54fcc284a0fb"} Jul 13 14:31:58.908409 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:31:58.908360 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" podStartSLOduration=1.395055989 podStartE2EDuration="1.908345479s" totalImagesPullingTime="513.28949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:31:58.906463293 +0000 UTC m=+660.590883600" watchObservedRunningTime="2026-07-13 14:31:58.908345479 +0000 UTC m=+660.592765800" Jul 13 14:32:04.198924 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:04.198893 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:32:05.477972 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:05.477938 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:32:05.768619 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:05.768541 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:32:07.710529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:07.710496 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:32:07.710945 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:07.710586 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:32:07.722666 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:07.722637 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:32:07.934966 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:07.934939 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-rrqrz" Jul 13 14:32:11.486472 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.486442 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n"] Jul 13 14:32:11.491359 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.491336 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.494099 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.494079 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 13 14:32:11.500325 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.500289 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n"] Jul 13 14:32:11.516173 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.516141 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:32:11.563444 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.563410 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0648a048-bf8d-4f7d-b5a3-237506a491f0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.563574 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.563476 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0648a048-bf8d-4f7d-b5a3-237506a491f0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.563574 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.563524 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0648a048-bf8d-4f7d-b5a3-237506a491f0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.563652 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.563601 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p279t\" (UniqueName: \"kubernetes.io/projected/0648a048-bf8d-4f7d-b5a3-237506a491f0-kube-api-access-p279t\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.563652 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.563640 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0648a048-bf8d-4f7d-b5a3-237506a491f0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.563720 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.563657 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0648a048-bf8d-4f7d-b5a3-237506a491f0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.664509 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.664476 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p279t\" (UniqueName: \"kubernetes.io/projected/0648a048-bf8d-4f7d-b5a3-237506a491f0-kube-api-access-p279t\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.664509 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.664515 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0648a048-bf8d-4f7d-b5a3-237506a491f0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.664750 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.664533 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0648a048-bf8d-4f7d-b5a3-237506a491f0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.664750 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.664574 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0648a048-bf8d-4f7d-b5a3-237506a491f0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.664750 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.664600 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0648a048-bf8d-4f7d-b5a3-237506a491f0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.664750 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.664643 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0648a048-bf8d-4f7d-b5a3-237506a491f0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.665101 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.665062 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0648a048-bf8d-4f7d-b5a3-237506a491f0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.665101 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.665086 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0648a048-bf8d-4f7d-b5a3-237506a491f0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.665256 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.665127 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0648a048-bf8d-4f7d-b5a3-237506a491f0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.666933 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.666903 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0648a048-bf8d-4f7d-b5a3-237506a491f0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.667165 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.667150 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0648a048-bf8d-4f7d-b5a3-237506a491f0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.673882 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.673860 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p279t\" (UniqueName: \"kubernetes.io/projected/0648a048-bf8d-4f7d-b5a3-237506a491f0-kube-api-access-p279t\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-25c5n\" (UID: \"0648a048-bf8d-4f7d-b5a3-237506a491f0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.802998 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.802919 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:11.946900 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:11.946872 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n"] Jul 13 14:32:11.947913 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:32:11.947889 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0648a048_bf8d_4f7d_b5a3_237506a491f0.slice/crio-104ec118168571b6060c105e7d451ad8e78237b6ab7d47885b195b56a98193db WatchSource:0}: Error finding container 104ec118168571b6060c105e7d451ad8e78237b6ab7d47885b195b56a98193db: Status 404 returned error can't find the container with id 104ec118168571b6060c105e7d451ad8e78237b6ab7d47885b195b56a98193db Jul 13 14:32:12.955069 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:12.955037 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" event={"ID":"0648a048-bf8d-4f7d-b5a3-237506a491f0","Type":"ContainerStarted","Data":"1023a15ee6e386cca3e0b507271d18fa5610bbbb879df4994af878392d737f78"} Jul 13 14:32:12.955443 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:12.955094 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" event={"ID":"0648a048-bf8d-4f7d-b5a3-237506a491f0","Type":"ContainerStarted","Data":"104ec118168571b6060c105e7d451ad8e78237b6ab7d47885b195b56a98193db"} Jul 13 14:32:12.978494 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:12.978448 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" podStartSLOduration=1.756241218 podStartE2EDuration="1.978433986s" totalImagesPullingTime="222.192768ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:32:12.976205211 +0000 UTC m=+674.660625527" watchObservedRunningTime="2026-07-13 14:32:12.978433986 +0000 UTC m=+674.662854293" Jul 13 14:32:21.020840 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:21.020801 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:32:21.803610 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:21.803575 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:21.803813 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:21.803647 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:21.816151 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:21.816118 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:32:22.012772 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:32:22.012740 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-25c5n" Jul 13 14:33:13.790516 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:33:13.790433 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:33:14.064750 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:33:14.064718 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:33:14.153683 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:33:14.153649 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:33:15.347578 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:33:15.347528 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:33:15.483016 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:33:15.482983 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:34:28.701386 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:28.701310 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:34:28.808869 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:28.808828 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-786f5ff9c9-bwtzz"] Jul 13 14:34:28.809118 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:28.809094 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" podUID="35b97602-8c8b-4a92-90e2-e6e489f41db9" containerName="manager" containerID="cri-o://12f2fd3bd254a276debd75d486d33ad8ae12afd7d51d776385edcc7747498ea2" gracePeriod=30 Jul 13 14:34:29.075797 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.075757 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" Jul 13 14:34:29.170600 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.170564 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdgbm\" (UniqueName: \"kubernetes.io/projected/35b97602-8c8b-4a92-90e2-e6e489f41db9-kube-api-access-hdgbm\") pod \"35b97602-8c8b-4a92-90e2-e6e489f41db9\" (UID: \"35b97602-8c8b-4a92-90e2-e6e489f41db9\") " Jul 13 14:34:29.170801 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.170641 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35b97602-8c8b-4a92-90e2-e6e489f41db9-webhook-cert\") pod \"35b97602-8c8b-4a92-90e2-e6e489f41db9\" (UID: \"35b97602-8c8b-4a92-90e2-e6e489f41db9\") " Jul 13 14:34:29.172635 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.172597 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b97602-8c8b-4a92-90e2-e6e489f41db9-kube-api-access-hdgbm" (OuterVolumeSpecName: "kube-api-access-hdgbm") pod "35b97602-8c8b-4a92-90e2-e6e489f41db9" (UID: "35b97602-8c8b-4a92-90e2-e6e489f41db9"). InnerVolumeSpecName "kube-api-access-hdgbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:34:29.172760 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.172684 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b97602-8c8b-4a92-90e2-e6e489f41db9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "35b97602-8c8b-4a92-90e2-e6e489f41db9" (UID: "35b97602-8c8b-4a92-90e2-e6e489f41db9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:34:29.271365 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.271281 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdgbm\" (UniqueName: \"kubernetes.io/projected/35b97602-8c8b-4a92-90e2-e6e489f41db9-kube-api-access-hdgbm\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:34:29.271365 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.271312 2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/35b97602-8c8b-4a92-90e2-e6e489f41db9-webhook-cert\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:34:29.682365 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.682334 2566 generic.go:356] "Generic (PLEG): container finished" podID="35b97602-8c8b-4a92-90e2-e6e489f41db9" containerID="12f2fd3bd254a276debd75d486d33ad8ae12afd7d51d776385edcc7747498ea2" exitCode=0 Jul 13 14:34:29.682539 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.682381 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" Jul 13 14:34:29.682539 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.682384 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" event={"ID":"35b97602-8c8b-4a92-90e2-e6e489f41db9","Type":"ContainerDied","Data":"12f2fd3bd254a276debd75d486d33ad8ae12afd7d51d776385edcc7747498ea2"} Jul 13 14:34:29.682539 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.682507 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786f5ff9c9-bwtzz" event={"ID":"35b97602-8c8b-4a92-90e2-e6e489f41db9","Type":"ContainerDied","Data":"47585188bd93d95873e8b8821d996ed20024b84804316ecefe0376923b0c8e2e"} Jul 13 14:34:29.682642 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.682542 2566 scope.go:117] "RemoveContainer" containerID="12f2fd3bd254a276debd75d486d33ad8ae12afd7d51d776385edcc7747498ea2" Jul 13 14:34:29.692035 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.692015 2566 scope.go:117] "RemoveContainer" containerID="12f2fd3bd254a276debd75d486d33ad8ae12afd7d51d776385edcc7747498ea2" Jul 13 14:34:29.692284 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:34:29.692262 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f2fd3bd254a276debd75d486d33ad8ae12afd7d51d776385edcc7747498ea2\": container with ID starting with 12f2fd3bd254a276debd75d486d33ad8ae12afd7d51d776385edcc7747498ea2 not found: ID does not exist" containerID="12f2fd3bd254a276debd75d486d33ad8ae12afd7d51d776385edcc7747498ea2" Jul 13 14:34:29.692360 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.692292 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f2fd3bd254a276debd75d486d33ad8ae12afd7d51d776385edcc7747498ea2"} err="failed to get container status \"12f2fd3bd254a276debd75d486d33ad8ae12afd7d51d776385edcc7747498ea2\": rpc error: code = NotFound desc = could not find container \"12f2fd3bd254a276debd75d486d33ad8ae12afd7d51d776385edcc7747498ea2\": container with ID starting with 12f2fd3bd254a276debd75d486d33ad8ae12afd7d51d776385edcc7747498ea2 not found: ID does not exist" Jul 13 14:34:29.712533 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.712508 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-786f5ff9c9-bwtzz"] Jul 13 14:34:29.714580 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:29.714560 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-786f5ff9c9-bwtzz"] Jul 13 14:34:30.520998 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.520962 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:34:30.686533 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.686504 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-786f5ff9c9-swdh4"] Jul 13 14:34:30.686960 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.686942 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b97602-8c8b-4a92-90e2-e6e489f41db9" containerName="manager" Jul 13 14:34:30.686960 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.686958 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b97602-8c8b-4a92-90e2-e6e489f41db9" containerName="manager" Jul 13 14:34:30.687149 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.687021 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b97602-8c8b-4a92-90e2-e6e489f41db9" containerName="manager" Jul 13 14:34:30.691380 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.691358 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786f5ff9c9-swdh4" Jul 13 14:34:30.695839 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.695817 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 13 14:34:30.696802 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.696765 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nlvgx\"" Jul 13 14:34:30.696921 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.696772 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 13 14:34:30.707218 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.707191 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-786f5ff9c9-swdh4"] Jul 13 14:34:30.783431 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.783338 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk4zj\" (UniqueName: \"kubernetes.io/projected/b23f893c-bb7f-4e48-9bfc-d538e2ffadc5-kube-api-access-zk4zj\") pod \"maas-controller-786f5ff9c9-swdh4\" (UID: \"b23f893c-bb7f-4e48-9bfc-d538e2ffadc5\") " pod="opendatahub/maas-controller-786f5ff9c9-swdh4" Jul 13 14:34:30.783776 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.783462 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b23f893c-bb7f-4e48-9bfc-d538e2ffadc5-webhook-cert\") pod \"maas-controller-786f5ff9c9-swdh4\" (UID: \"b23f893c-bb7f-4e48-9bfc-d538e2ffadc5\") " pod="opendatahub/maas-controller-786f5ff9c9-swdh4" Jul 13 14:34:30.867883 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.867854 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35b97602-8c8b-4a92-90e2-e6e489f41db9" path="/var/lib/kubelet/pods/35b97602-8c8b-4a92-90e2-e6e489f41db9/volumes" Jul 13 14:34:30.884225 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.884202 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b23f893c-bb7f-4e48-9bfc-d538e2ffadc5-webhook-cert\") pod \"maas-controller-786f5ff9c9-swdh4\" (UID: \"b23f893c-bb7f-4e48-9bfc-d538e2ffadc5\") " pod="opendatahub/maas-controller-786f5ff9c9-swdh4" Jul 13 14:34:30.884371 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.884282 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk4zj\" (UniqueName: \"kubernetes.io/projected/b23f893c-bb7f-4e48-9bfc-d538e2ffadc5-kube-api-access-zk4zj\") pod \"maas-controller-786f5ff9c9-swdh4\" (UID: \"b23f893c-bb7f-4e48-9bfc-d538e2ffadc5\") " pod="opendatahub/maas-controller-786f5ff9c9-swdh4" Jul 13 14:34:30.886523 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.886506 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b23f893c-bb7f-4e48-9bfc-d538e2ffadc5-webhook-cert\") pod \"maas-controller-786f5ff9c9-swdh4\" (UID: \"b23f893c-bb7f-4e48-9bfc-d538e2ffadc5\") " pod="opendatahub/maas-controller-786f5ff9c9-swdh4" Jul 13 14:34:30.896423 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:30.896397 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk4zj\" (UniqueName: \"kubernetes.io/projected/b23f893c-bb7f-4e48-9bfc-d538e2ffadc5-kube-api-access-zk4zj\") pod \"maas-controller-786f5ff9c9-swdh4\" (UID: \"b23f893c-bb7f-4e48-9bfc-d538e2ffadc5\") " pod="opendatahub/maas-controller-786f5ff9c9-swdh4" Jul 13 14:34:31.001652 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:31.001613 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-786f5ff9c9-swdh4" Jul 13 14:34:31.149335 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:31.149308 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-786f5ff9c9-swdh4"] Jul 13 14:34:31.152009 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:34:31.151977 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23f893c_bb7f_4e48_9bfc_d538e2ffadc5.slice/crio-d2f4eaa5d08af87774c448e67e93ed9fd4d18903040f4bf0f0e2999fa1db67a9 WatchSource:0}: Error finding container d2f4eaa5d08af87774c448e67e93ed9fd4d18903040f4bf0f0e2999fa1db67a9: Status 404 returned error can't find the container with id d2f4eaa5d08af87774c448e67e93ed9fd4d18903040f4bf0f0e2999fa1db67a9 Jul 13 14:34:31.696373 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:31.696335 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786f5ff9c9-swdh4" event={"ID":"b23f893c-bb7f-4e48-9bfc-d538e2ffadc5","Type":"ContainerStarted","Data":"f646881028b1072fba5341202327b99fbd2d4843cf2c9b9285c8960d270a13e3"} Jul 13 14:34:31.696529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:31.696400 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-786f5ff9c9-swdh4" event={"ID":"b23f893c-bb7f-4e48-9bfc-d538e2ffadc5","Type":"ContainerStarted","Data":"d2f4eaa5d08af87774c448e67e93ed9fd4d18903040f4bf0f0e2999fa1db67a9"} Jul 13 14:34:31.727980 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:31.727918 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-786f5ff9c9-swdh4" podStartSLOduration=1.284866954 podStartE2EDuration="1.72790336s" totalImagesPullingTime="443.036406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:34:31.725705691 +0000 UTC m=+813.410125997" watchObservedRunningTime="2026-07-13 14:34:31.72790336 +0000 UTC m=+813.412323666" Jul 13 14:34:32.702049 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:32.702022 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-786f5ff9c9-swdh4" Jul 13 14:34:43.714032 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:34:43.713999 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-786f5ff9c9-swdh4" Jul 13 14:35:11.515017 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:35:11.514983 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:35:11.590451 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:35:11.590419 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:35:17.838983 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:35:17.838950 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:35:17.901691 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:35:17.901663 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:35:17.980871 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:35:17.980842 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:35:18.080348 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:35:18.080317 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:35:18.245690 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:35:18.245603 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:35:24.776719 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:35:24.776686 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:35:24.842003 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:35:24.841969 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:36:29.968523 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:36:29.968488 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:37:45.454804 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:37:45.454756 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:37:45.548906 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:37:45.548871 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:39:03.882221 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:03.882142 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:39:24.514836 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:24.514802 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld"] Jul 13 14:39:24.515246 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:24.515004 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" podUID="4d3b35e1-204a-492f-b894-0ef94c7f8c1e" containerName="manager" containerID="cri-o://09352639405ef8794a46ab6e80528481469383dda5c7f710f7ebda69981acf0a" gracePeriod=10 Jul 13 14:39:27.971901 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:27.971874 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" Jul 13 14:39:28.120093 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.120012 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndfx6\" (UniqueName: \"kubernetes.io/projected/4d3b35e1-204a-492f-b894-0ef94c7f8c1e-kube-api-access-ndfx6\") pod \"4d3b35e1-204a-492f-b894-0ef94c7f8c1e\" (UID: \"4d3b35e1-204a-492f-b894-0ef94c7f8c1e\") " Jul 13 14:39:28.120233 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.120110 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d3b35e1-204a-492f-b894-0ef94c7f8c1e-extensions-socket-volume\") pod \"4d3b35e1-204a-492f-b894-0ef94c7f8c1e\" (UID: \"4d3b35e1-204a-492f-b894-0ef94c7f8c1e\") " Jul 13 14:39:28.120489 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.120462 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3b35e1-204a-492f-b894-0ef94c7f8c1e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4d3b35e1-204a-492f-b894-0ef94c7f8c1e" (UID: "4d3b35e1-204a-492f-b894-0ef94c7f8c1e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:39:28.121981 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.121953 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3b35e1-204a-492f-b894-0ef94c7f8c1e-kube-api-access-ndfx6" (OuterVolumeSpecName: "kube-api-access-ndfx6") pod "4d3b35e1-204a-492f-b894-0ef94c7f8c1e" (UID: "4d3b35e1-204a-492f-b894-0ef94c7f8c1e"). InnerVolumeSpecName "kube-api-access-ndfx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:39:28.221027 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.221002 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d3b35e1-204a-492f-b894-0ef94c7f8c1e-extensions-socket-volume\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:39:28.221027 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.221022 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndfx6\" (UniqueName: \"kubernetes.io/projected/4d3b35e1-204a-492f-b894-0ef94c7f8c1e-kube-api-access-ndfx6\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:39:28.283913 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.283877 2566 generic.go:356] "Generic (PLEG): container finished" podID="4d3b35e1-204a-492f-b894-0ef94c7f8c1e" containerID="09352639405ef8794a46ab6e80528481469383dda5c7f710f7ebda69981acf0a" exitCode=0 Jul 13 14:39:28.284062 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.283919 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" Jul 13 14:39:28.284062 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.283961 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" event={"ID":"4d3b35e1-204a-492f-b894-0ef94c7f8c1e","Type":"ContainerDied","Data":"09352639405ef8794a46ab6e80528481469383dda5c7f710f7ebda69981acf0a"} Jul 13 14:39:28.284062 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.284006 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld" event={"ID":"4d3b35e1-204a-492f-b894-0ef94c7f8c1e","Type":"ContainerDied","Data":"c791a6ef3c1be02bc20f6f48e3b6381a8e262b31adf6465b38922f7d4c57febb"} Jul 13 14:39:28.284062 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.284027 2566 scope.go:117] "RemoveContainer" containerID="09352639405ef8794a46ab6e80528481469383dda5c7f710f7ebda69981acf0a" Jul 13 14:39:28.293777 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.293758 2566 scope.go:117] "RemoveContainer" containerID="09352639405ef8794a46ab6e80528481469383dda5c7f710f7ebda69981acf0a" Jul 13 14:39:28.294051 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:39:28.294033 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09352639405ef8794a46ab6e80528481469383dda5c7f710f7ebda69981acf0a\": container with ID starting with 09352639405ef8794a46ab6e80528481469383dda5c7f710f7ebda69981acf0a not found: ID does not exist" containerID="09352639405ef8794a46ab6e80528481469383dda5c7f710f7ebda69981acf0a" Jul 13 14:39:28.294116 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.294065 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09352639405ef8794a46ab6e80528481469383dda5c7f710f7ebda69981acf0a"} err="failed to get container status \"09352639405ef8794a46ab6e80528481469383dda5c7f710f7ebda69981acf0a\": rpc error: code = NotFound desc = could not find container \"09352639405ef8794a46ab6e80528481469383dda5c7f710f7ebda69981acf0a\": container with ID starting with 09352639405ef8794a46ab6e80528481469383dda5c7f710f7ebda69981acf0a not found: ID does not exist" Jul 13 14:39:28.308081 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.308059 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld"] Jul 13 14:39:28.310354 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.310332 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-m8hld"] Jul 13 14:39:28.867374 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:28.867342 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d3b35e1-204a-492f-b894-0ef94c7f8c1e" path="/var/lib/kubelet/pods/4d3b35e1-204a-492f-b894-0ef94c7f8c1e/volumes" Jul 13 14:39:36.454696 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:39:36.454668 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:40:25.753150 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:25.753070 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n"] Jul 13 14:40:25.753546 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:25.753468 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d3b35e1-204a-492f-b894-0ef94c7f8c1e" containerName="manager" Jul 13 14:40:25.753546 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:25.753481 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d3b35e1-204a-492f-b894-0ef94c7f8c1e" containerName="manager" Jul 13 14:40:25.753620 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:25.753550 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d3b35e1-204a-492f-b894-0ef94c7f8c1e" containerName="manager" Jul 13 14:40:25.756500 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:25.756481 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n" Jul 13 14:40:25.759322 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:25.759302 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-clhnl\"" Jul 13 14:40:25.769522 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:25.769499 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n"] Jul 13 14:40:25.820693 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:25.820666 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh5ff\" (UniqueName: \"kubernetes.io/projected/2f403521-759c-46c6-afee-9d79c8267d01-kube-api-access-wh5ff\") pod \"kuadrant-operator-controller-manager-86878c769f-fpb6n\" (UID: \"2f403521-759c-46c6-afee-9d79c8267d01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n" Jul 13 14:40:25.820874 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:25.820701 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f403521-759c-46c6-afee-9d79c8267d01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fpb6n\" (UID: \"2f403521-759c-46c6-afee-9d79c8267d01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n" Jul 13 14:40:25.921902 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:25.921871 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f403521-759c-46c6-afee-9d79c8267d01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fpb6n\" (UID: \"2f403521-759c-46c6-afee-9d79c8267d01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n" Jul 13 14:40:25.922047 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:25.922019 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh5ff\" (UniqueName: \"kubernetes.io/projected/2f403521-759c-46c6-afee-9d79c8267d01-kube-api-access-wh5ff\") pod \"kuadrant-operator-controller-manager-86878c769f-fpb6n\" (UID: \"2f403521-759c-46c6-afee-9d79c8267d01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n" Jul 13 14:40:25.922321 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:25.922300 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f403521-759c-46c6-afee-9d79c8267d01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-fpb6n\" (UID: \"2f403521-759c-46c6-afee-9d79c8267d01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n" Jul 13 14:40:25.931391 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:25.931371 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh5ff\" (UniqueName: \"kubernetes.io/projected/2f403521-759c-46c6-afee-9d79c8267d01-kube-api-access-wh5ff\") pod \"kuadrant-operator-controller-manager-86878c769f-fpb6n\" (UID: \"2f403521-759c-46c6-afee-9d79c8267d01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n" Jul 13 14:40:26.066468 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:26.066433 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n" Jul 13 14:40:26.205443 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:26.205420 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n"] Jul 13 14:40:26.206845 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:40:26.206820 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f403521_759c_46c6_afee_9d79c8267d01.slice/crio-8afaef22903b5bcc5dfe6a78ae383ef744d4153b9a1f7919f4570353bb055181 WatchSource:0}: Error finding container 8afaef22903b5bcc5dfe6a78ae383ef744d4153b9a1f7919f4570353bb055181: Status 404 returned error can't find the container with id 8afaef22903b5bcc5dfe6a78ae383ef744d4153b9a1f7919f4570353bb055181 Jul 13 14:40:26.583083 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:26.583044 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n" event={"ID":"2f403521-759c-46c6-afee-9d79c8267d01","Type":"ContainerStarted","Data":"95906a0788f57a76960522e8a6c59a96c6817bb9705715c039f223e6e2f453aa"} Jul 13 14:40:26.583252 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:26.583106 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n" event={"ID":"2f403521-759c-46c6-afee-9d79c8267d01","Type":"ContainerStarted","Data":"8afaef22903b5bcc5dfe6a78ae383ef744d4153b9a1f7919f4570353bb055181"} Jul 13 14:40:26.583252 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:26.583156 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n" Jul 13 14:40:26.609259 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:26.609215 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n" podStartSLOduration=1.609195135 podStartE2EDuration="1.609195135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:40:26.606610654 +0000 UTC m=+1168.291030962" watchObservedRunningTime="2026-07-13 14:40:26.609195135 +0000 UTC m=+1168.293615440" Jul 13 14:40:37.589843 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:40:37.589813 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-fpb6n" Jul 13 14:41:26.455169 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:41:26.455135 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:43:02.582880 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:43:02.582843 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:43:08.120126 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:43:08.120087 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:45:00.154833 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:45:00.154728 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj"] Jul 13 14:45:00.158513 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:45:00.158490 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj" Jul 13 14:45:00.161243 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:45:00.161223 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fkppd\"" Jul 13 14:45:00.167838 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:45:00.167822 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj" Jul 13 14:45:00.181583 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:45:00.181557 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj"] Jul 13 14:45:00.306433 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:45:00.306404 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj"] Jul 13 14:45:00.307987 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:45:00.307961 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ecbfe0_6ca8_48ba_ab13_af4f9f3e4c21.slice/crio-5b5616bbe464d71d037e00f7d9bc9f83459a72694e28a43e9306b902e20f3256 WatchSource:0}: Error finding container 5b5616bbe464d71d037e00f7d9bc9f83459a72694e28a43e9306b902e20f3256: Status 404 returned error can't find the container with id 5b5616bbe464d71d037e00f7d9bc9f83459a72694e28a43e9306b902e20f3256 Jul 13 14:45:00.309771 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:45:00.309754 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 14:45:01.037107 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:45:01.037073 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj" event={"ID":"b9ecbfe0-6ca8-48ba-ab13-af4f9f3e4c21","Type":"ContainerStarted","Data":"5b5616bbe464d71d037e00f7d9bc9f83459a72694e28a43e9306b902e20f3256"} Jul 13 14:45:03.050409 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:45:03.050318 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj" event={"ID":"b9ecbfe0-6ca8-48ba-ab13-af4f9f3e4c21","Type":"ContainerStarted","Data":"b47f1e59978de542de8658480aa274a772deff0e8d4a24d02601b9a7ea92d7e1"} Jul 13 14:45:03.073591 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:45:03.073510 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj" podStartSLOduration=0.74442989 podStartE2EDuration="3.073491463s" totalImagesPullingTime="2.329061573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:45:03.070345223 +0000 UTC m=+1444.754765528" watchObservedRunningTime="2026-07-13 14:45:03.073491463 +0000 UTC m=+1444.757911770" Jul 13 14:45:53.649914 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:45:53.649877 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:00.013391 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:00.013357 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj"] Jul 13 14:46:00.013883 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:00.013535 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj" podUID="b9ecbfe0-6ca8-48ba-ab13-af4f9f3e4c21" containerName="cleanup" containerID="cri-o://b47f1e59978de542de8658480aa274a772deff0e8d4a24d02601b9a7ea92d7e1" gracePeriod=30 Jul 13 14:46:20.928873 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:20.928766 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:24.559091 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:24.559064 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:30.203078 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:30.203046 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj" Jul 13 14:46:30.524509 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:30.524475 2566 generic.go:356] "Generic (PLEG): container finished" podID="b9ecbfe0-6ca8-48ba-ab13-af4f9f3e4c21" containerID="b47f1e59978de542de8658480aa274a772deff0e8d4a24d02601b9a7ea92d7e1" exitCode=137 Jul 13 14:46:30.524688 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:30.524518 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj" Jul 13 14:46:30.524688 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:30.524522 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj" event={"ID":"b9ecbfe0-6ca8-48ba-ab13-af4f9f3e4c21","Type":"ContainerDied","Data":"b47f1e59978de542de8658480aa274a772deff0e8d4a24d02601b9a7ea92d7e1"} Jul 13 14:46:30.524688 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:30.524564 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj" event={"ID":"b9ecbfe0-6ca8-48ba-ab13-af4f9f3e4c21","Type":"ContainerDied","Data":"5b5616bbe464d71d037e00f7d9bc9f83459a72694e28a43e9306b902e20f3256"} Jul 13 14:46:30.524688 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:30.524586 2566 scope.go:117] "RemoveContainer" containerID="b47f1e59978de542de8658480aa274a772deff0e8d4a24d02601b9a7ea92d7e1" Jul 13 14:46:30.534450 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:30.534426 2566 scope.go:117] "RemoveContainer" containerID="b47f1e59978de542de8658480aa274a772deff0e8d4a24d02601b9a7ea92d7e1" Jul 13 14:46:30.534751 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:46:30.534729 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47f1e59978de542de8658480aa274a772deff0e8d4a24d02601b9a7ea92d7e1\": container with ID starting with b47f1e59978de542de8658480aa274a772deff0e8d4a24d02601b9a7ea92d7e1 not found: ID does not exist" containerID="b47f1e59978de542de8658480aa274a772deff0e8d4a24d02601b9a7ea92d7e1" Jul 13 14:46:30.534862 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:30.534762 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47f1e59978de542de8658480aa274a772deff0e8d4a24d02601b9a7ea92d7e1"} err="failed to get container status \"b47f1e59978de542de8658480aa274a772deff0e8d4a24d02601b9a7ea92d7e1\": rpc error: code = NotFound desc = could not find container \"b47f1e59978de542de8658480aa274a772deff0e8d4a24d02601b9a7ea92d7e1\": container with ID starting with b47f1e59978de542de8658480aa274a772deff0e8d4a24d02601b9a7ea92d7e1 not found: ID does not exist" Jul 13 14:46:30.551530 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:30.551500 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj"] Jul 13 14:46:30.556408 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:30.556382 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29732565-xwrlj"] Jul 13 14:46:30.867174 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:30.867141 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9ecbfe0-6ca8-48ba-ab13-af4f9f3e4c21" path="/var/lib/kubelet/pods/b9ecbfe0-6ca8-48ba-ab13-af4f9f3e4c21/volumes" Jul 13 14:46:31.630320 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:31.630268 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:32.148138 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:32.148111 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:40.407174 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:40.407140 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:46:45.526028 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:46:45.525991 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:48:44.299885 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:48:44.299846 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:48:55.418361 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:48:55.418323 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:48:55.503841 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:48:55.503814 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:48.254226 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:52:48.254187 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:57.693394 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:52:57.693357 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:58.910419 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:52:58.910383 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:52:59.008502 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:52:59.008467 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:04.507046 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.506998 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:04.579986 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.579949 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:04.714739 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.714716 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97"] Jul 13 14:53:04.715208 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.715193 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ecbfe0-6ca8-48ba-ab13-af4f9f3e4c21" containerName="cleanup" Jul 13 14:53:04.715257 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.715210 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ecbfe0-6ca8-48ba-ab13-af4f9f3e4c21" containerName="cleanup" Jul 13 14:53:04.715296 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.715275 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9ecbfe0-6ca8-48ba-ab13-af4f9f3e4c21" containerName="cleanup" Jul 13 14:53:04.718248 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.718233 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.720949 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.720931 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ccb2daa0-openshift-default-dockercfg-kfjvg\"" Jul 13 14:53:04.746911 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.746880 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97"] Jul 13 14:53:04.748854 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.748828 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:04.821184 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.821147 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:04.834694 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.834660 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-token\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.834853 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.834733 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m9dj\" (UniqueName: \"kubernetes.io/projected/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-kube-api-access-6m9dj\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.834853 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.834769 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-workload-socket\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.834951 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.834862 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-podinfo\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.834951 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.834914 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-envoy\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.835025 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.834966 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-workload-certs\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.835025 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.834993 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-data\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.835109 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.835023 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istiod-ca-cert\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.835109 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.835080 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-credential-socket\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.936468 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.936436 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-token\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.936661 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.936488 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m9dj\" (UniqueName: \"kubernetes.io/projected/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-kube-api-access-6m9dj\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.936661 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.936510 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-workload-socket\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.936661 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.936630 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-podinfo\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.936862 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.936689 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-envoy\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.936862 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.936749 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-workload-certs\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.936862 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.936773 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-data\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.936862 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.936829 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istiod-ca-cert\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.937074 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.936873 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-workload-socket\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.937074 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.936885 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-credential-socket\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.937177 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.937120 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-workload-certs\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.937177 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.937163 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-data\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.937325 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.937303 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-credential-socket\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.937750 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.937726 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istiod-ca-cert\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.938875 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.938848 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-envoy\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.939126 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.939109 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-podinfo\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.945132 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.945107 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-token\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:04.945314 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:04.945296 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m9dj\" (UniqueName: \"kubernetes.io/projected/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-kube-api-access-6m9dj\") pod \"e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:05.028331 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:05.028300 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:05.166815 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:05.166776 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97"] Jul 13 14:53:05.168156 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:53:05.168127 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf751f9_96aa_49c8_ad3f_90c9dc54a799.slice/crio-18618ecc78ac58c2a063fbe2e82b7a400cbaa903677ac347de4b26900c875140 WatchSource:0}: Error finding container 18618ecc78ac58c2a063fbe2e82b7a400cbaa903677ac347de4b26900c875140: Status 404 returned error can't find the container with id 18618ecc78ac58c2a063fbe2e82b7a400cbaa903677ac347de4b26900c875140 Jul 13 14:53:05.170466 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:05.170438 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:53:05.170533 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:05.170498 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:53:05.170533 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:05.170526 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:53:05.573026 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:05.572995 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" event={"ID":"3cf751f9-96aa-49c8-ad3f-90c9dc54a799","Type":"ContainerStarted","Data":"b740adeb055cf8185ebcdaa3bde8449a5d0bbb712e983c0c1f4d124a14a836c5"} Jul 13 14:53:05.573388 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:05.573052 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" event={"ID":"3cf751f9-96aa-49c8-ad3f-90c9dc54a799","Type":"ContainerStarted","Data":"18618ecc78ac58c2a063fbe2e82b7a400cbaa903677ac347de4b26900c875140"} Jul 13 14:53:05.599742 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:05.599697 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" podStartSLOduration=1.599681146 podStartE2EDuration="1.599681146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:05.598620325 +0000 UTC m=+1927.283040630" watchObservedRunningTime="2026-07-13 14:53:05.599681146 +0000 UTC m=+1927.284101451" Jul 13 14:53:06.028687 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:06.028595 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:06.034546 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:06.034519 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:06.578768 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:06.578733 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:06.579853 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:06.579834 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:07.209661 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:07.209616 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:07.308662 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:07.308626 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:08.305503 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:08.305464 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:11.081182 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:11.081150 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:11.227639 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:11.227605 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97"] Jul 13 14:53:11.227849 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:11.227777 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" podUID="3cf751f9-96aa-49c8-ad3f-90c9dc54a799" containerName="istio-proxy" containerID="cri-o://b740adeb055cf8185ebcdaa3bde8449a5d0bbb712e983c0c1f4d124a14a836c5" gracePeriod=30 Jul 13 14:53:11.249810 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:11.249755 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:11.361216 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:11.361121 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:12.585849 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:12.585813 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:16.491294 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.491260 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:16.544498 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.544402 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-data\") pod \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " Jul 13 14:53:16.544498 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.544465 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m9dj\" (UniqueName: \"kubernetes.io/projected/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-kube-api-access-6m9dj\") pod \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " Jul 13 14:53:16.544498 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.544491 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-envoy\") pod \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " Jul 13 14:53:16.544812 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.544520 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-token\") pod \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " Jul 13 14:53:16.544812 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.544545 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-workload-certs\") pod \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " Jul 13 14:53:16.544812 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.544595 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istiod-ca-cert\") pod \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " Jul 13 14:53:16.544812 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.544624 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-workload-socket\") pod \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " Jul 13 14:53:16.544812 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.544659 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-credential-socket\") pod \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " Jul 13 14:53:16.544812 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.544687 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-podinfo\") pod \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\" (UID: \"3cf751f9-96aa-49c8-ad3f-90c9dc54a799\") " Jul 13 14:53:16.544812 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.544713 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-data" (OuterVolumeSpecName: "istio-data") pod "3cf751f9-96aa-49c8-ad3f-90c9dc54a799" (UID: "3cf751f9-96aa-49c8-ad3f-90c9dc54a799"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:16.545202 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.545003 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-data\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:16.545202 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.545059 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3cf751f9-96aa-49c8-ad3f-90c9dc54a799" (UID: "3cf751f9-96aa-49c8-ad3f-90c9dc54a799"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:53:16.545320 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.545219 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3cf751f9-96aa-49c8-ad3f-90c9dc54a799" (UID: "3cf751f9-96aa-49c8-ad3f-90c9dc54a799"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:16.545375 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.545345 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3cf751f9-96aa-49c8-ad3f-90c9dc54a799" (UID: "3cf751f9-96aa-49c8-ad3f-90c9dc54a799"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:16.545523 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.545484 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3cf751f9-96aa-49c8-ad3f-90c9dc54a799" (UID: "3cf751f9-96aa-49c8-ad3f-90c9dc54a799"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:16.547016 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.546993 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3cf751f9-96aa-49c8-ad3f-90c9dc54a799" (UID: "3cf751f9-96aa-49c8-ad3f-90c9dc54a799"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 14:53:16.547237 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.547194 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3cf751f9-96aa-49c8-ad3f-90c9dc54a799" (UID: "3cf751f9-96aa-49c8-ad3f-90c9dc54a799"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:16.547237 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.547212 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-token" (OuterVolumeSpecName: "istio-token") pod "3cf751f9-96aa-49c8-ad3f-90c9dc54a799" (UID: "3cf751f9-96aa-49c8-ad3f-90c9dc54a799"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:53:16.547407 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.547354 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-kube-api-access-6m9dj" (OuterVolumeSpecName: "kube-api-access-6m9dj") pod "3cf751f9-96aa-49c8-ad3f-90c9dc54a799" (UID: "3cf751f9-96aa-49c8-ad3f-90c9dc54a799"). InnerVolumeSpecName "kube-api-access-6m9dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:53:16.630587 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.630551 2566 generic.go:356] "Generic (PLEG): container finished" podID="3cf751f9-96aa-49c8-ad3f-90c9dc54a799" containerID="b740adeb055cf8185ebcdaa3bde8449a5d0bbb712e983c0c1f4d124a14a836c5" exitCode=0 Jul 13 14:53:16.630755 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.630596 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" Jul 13 14:53:16.630755 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.630638 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" event={"ID":"3cf751f9-96aa-49c8-ad3f-90c9dc54a799","Type":"ContainerDied","Data":"b740adeb055cf8185ebcdaa3bde8449a5d0bbb712e983c0c1f4d124a14a836c5"} Jul 13 14:53:16.630755 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.630690 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97" event={"ID":"3cf751f9-96aa-49c8-ad3f-90c9dc54a799","Type":"ContainerDied","Data":"18618ecc78ac58c2a063fbe2e82b7a400cbaa903677ac347de4b26900c875140"} Jul 13 14:53:16.630755 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.630714 2566 scope.go:117] "RemoveContainer" containerID="b740adeb055cf8185ebcdaa3bde8449a5d0bbb712e983c0c1f4d124a14a836c5" Jul 13 14:53:16.641285 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.641267 2566 scope.go:117] "RemoveContainer" containerID="b740adeb055cf8185ebcdaa3bde8449a5d0bbb712e983c0c1f4d124a14a836c5" Jul 13 14:53:16.641562 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:53:16.641543 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b740adeb055cf8185ebcdaa3bde8449a5d0bbb712e983c0c1f4d124a14a836c5\": container with ID starting with b740adeb055cf8185ebcdaa3bde8449a5d0bbb712e983c0c1f4d124a14a836c5 not found: ID does not exist" containerID="b740adeb055cf8185ebcdaa3bde8449a5d0bbb712e983c0c1f4d124a14a836c5" Jul 13 14:53:16.641636 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.641574 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b740adeb055cf8185ebcdaa3bde8449a5d0bbb712e983c0c1f4d124a14a836c5"} err="failed to get container status \"b740adeb055cf8185ebcdaa3bde8449a5d0bbb712e983c0c1f4d124a14a836c5\": rpc error: code = NotFound desc = could not find container \"b740adeb055cf8185ebcdaa3bde8449a5d0bbb712e983c0c1f4d124a14a836c5\": container with ID starting with b740adeb055cf8185ebcdaa3bde8449a5d0bbb712e983c0c1f4d124a14a836c5 not found: ID does not exist" Jul 13 14:53:16.645874 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.645853 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-workload-certs\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:16.645954 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.645876 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istiod-ca-cert\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:16.645954 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.645884 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-workload-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:16.645954 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.645892 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-credential-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:16.645954 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.645902 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-podinfo\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:16.645954 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.645911 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m9dj\" (UniqueName: \"kubernetes.io/projected/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-kube-api-access-6m9dj\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:16.645954 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.645919 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-envoy\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:16.645954 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.645927 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf751f9-96aa-49c8-ad3f-90c9dc54a799-istio-token\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:16.655757 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.655733 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97"] Jul 13 14:53:16.660528 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.660507 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ccb2daa0-openshift-default-88bcfdcc5-fqs97"] Jul 13 14:53:16.868301 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:16.868265 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cf751f9-96aa-49c8-ad3f-90c9dc54a799" path="/var/lib/kubelet/pods/3cf751f9-96aa-49c8-ad3f-90c9dc54a799/volumes" Jul 13 14:53:17.838941 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:17.838909 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:17.903760 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:17.903725 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:18.025115 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.025082 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq"] Jul 13 14:53:18.025465 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.025453 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cf751f9-96aa-49c8-ad3f-90c9dc54a799" containerName="istio-proxy" Jul 13 14:53:18.025511 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.025467 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cf751f9-96aa-49c8-ad3f-90c9dc54a799" containerName="istio-proxy" Jul 13 14:53:18.025543 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.025536 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cf751f9-96aa-49c8-ad3f-90c9dc54a799" containerName="istio-proxy" Jul 13 14:53:18.029983 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.029951 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.034808 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.034756 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d0ab5f95-openshift-default-dockercfg-pdplq\"" Jul 13 14:53:18.043630 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.043604 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq"] Jul 13 14:53:18.057971 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.057940 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-token\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.057971 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.057970 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqpp\" (UniqueName: \"kubernetes.io/projected/01cbfaba-aba5-4eca-bea6-2f1ac084f144-kube-api-access-bgqpp\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.058187 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.058008 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-podinfo\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.058187 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.058043 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-data\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.058187 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.058093 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-workload-socket\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.058187 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.058119 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-envoy\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.058187 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.058137 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-credential-socket\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.058464 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.058194 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istiod-ca-cert\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.058464 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.058219 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-workload-certs\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.088674 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.088640 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:18.159234 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.159146 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-workload-socket\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.159234 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.159184 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-envoy\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.159234 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.159206 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-credential-socket\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.159510 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.159245 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istiod-ca-cert\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.159510 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.159268 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-workload-certs\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.159510 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.159290 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-token\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.159510 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.159314 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqpp\" (UniqueName: \"kubernetes.io/projected/01cbfaba-aba5-4eca-bea6-2f1ac084f144-kube-api-access-bgqpp\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.159510 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.159353 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-podinfo\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.159510 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.159388 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-data\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.159822 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.159665 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-credential-socket\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.159822 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.159807 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-data\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.159922 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.159862 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-workload-socket\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.160069 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.160048 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-workload-certs\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.160112 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.160051 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istiod-ca-cert\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.161552 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.161533 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-envoy\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.161916 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.161898 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-podinfo\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.170975 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.170951 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-token\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.171436 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.171419 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqpp\" (UniqueName: \"kubernetes.io/projected/01cbfaba-aba5-4eca-bea6-2f1ac084f144-kube-api-access-bgqpp\") pod \"e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.189751 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.189717 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:18.342117 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.342082 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:18.493198 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.493165 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq"] Jul 13 14:53:18.494580 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:53:18.494557 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01cbfaba_aba5_4eca_bea6_2f1ac084f144.slice/crio-a1b9784db598291f1704959df6a097f9800b349b5bf03d3c86a050efc39f4317 WatchSource:0}: Error finding container a1b9784db598291f1704959df6a097f9800b349b5bf03d3c86a050efc39f4317: Status 404 returned error can't find the container with id a1b9784db598291f1704959df6a097f9800b349b5bf03d3c86a050efc39f4317 Jul 13 14:53:18.496553 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.496526 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:53:18.496617 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.496584 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:53:18.496655 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.496614 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:53:18.646659 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.646621 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" event={"ID":"01cbfaba-aba5-4eca-bea6-2f1ac084f144","Type":"ContainerStarted","Data":"35d687cf2d9d347cf0414b3da0fa831cc2441b541bbf984b2a40cca44b15701b"} Jul 13 14:53:18.646854 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.646685 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" event={"ID":"01cbfaba-aba5-4eca-bea6-2f1ac084f144","Type":"ContainerStarted","Data":"a1b9784db598291f1704959df6a097f9800b349b5bf03d3c86a050efc39f4317"} Jul 13 14:53:18.670860 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:18.670816 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" podStartSLOduration=0.670799622 podStartE2EDuration="670.799622ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:18.668966383 +0000 UTC m=+1940.353386690" watchObservedRunningTime="2026-07-13 14:53:18.670799622 +0000 UTC m=+1940.355219929" Jul 13 14:53:19.342991 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.342950 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:19.348179 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.348151 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:19.363757 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.363721 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:19.605092 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.605011 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:19.653886 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.653854 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:19.654732 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.654713 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:19.753842 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.753813 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w"] Jul 13 14:53:19.757542 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.757525 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" Jul 13 14:53:19.760437 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.760414 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d0ab5f95-serving-cert\"" Jul 13 14:53:19.760851 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.760835 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rxm62\"" Jul 13 14:53:19.771145 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.771119 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w"] Jul 13 14:53:19.851367 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.851338 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:19.874107 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.874039 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-openshift-service-ca\") pod \"maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w\" (UID: \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" Jul 13 14:53:19.874107 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.874100 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtsjq\" (UniqueName: \"kubernetes.io/projected/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-kube-api-access-dtsjq\") pod \"maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w\" (UID: \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" Jul 13 14:53:19.874264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.874173 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-maas-api-tls\") pod \"maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w\" (UID: \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" Jul 13 14:53:19.975151 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.975116 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-maas-api-tls\") pod \"maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w\" (UID: \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" Jul 13 14:53:19.975327 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.975161 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-openshift-service-ca\") pod \"maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w\" (UID: \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" Jul 13 14:53:19.975327 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:53:19.975262 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-serving-cert: secret "maas-api-e2e-ait-d0ab5f95-serving-cert" not found Jul 13 14:53:19.975327 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.975301 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtsjq\" (UniqueName: \"kubernetes.io/projected/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-kube-api-access-dtsjq\") pod \"maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w\" (UID: \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" Jul 13 14:53:19.975516 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:53:19.975345 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-maas-api-tls podName:ec9ce717-43b0-47a4-85d0-6348d24a7dd2 nodeName:}" failed. No retries permitted until 2026-07-13 14:53:20.47532343 +0000 UTC m=+1942.159743714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-maas-api-tls") pod "maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" (UID: "ec9ce717-43b0-47a4-85d0-6348d24a7dd2") : secret "maas-api-e2e-ait-d0ab5f95-serving-cert" not found Jul 13 14:53:19.975938 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.975920 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-openshift-service-ca\") pod \"maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w\" (UID: \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" Jul 13 14:53:19.990162 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:19.990138 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtsjq\" (UniqueName: \"kubernetes.io/projected/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-kube-api-access-dtsjq\") pod \"maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w\" (UID: \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" Jul 13 14:53:20.106003 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:20.105957 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:20.480221 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:20.480187 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-maas-api-tls\") pod \"maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w\" (UID: \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" Jul 13 14:53:20.482587 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:20.482557 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-maas-api-tls\") pod \"maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w\" (UID: \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" Jul 13 14:53:20.571904 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:20.571862 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:20.670624 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:20.670592 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" Jul 13 14:53:20.704518 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:20.704480 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:20.833635 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:20.833605 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w"] Jul 13 14:53:20.835356 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:53:20.835325 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9ce717_43b0_47a4_85d0_6348d24a7dd2.slice/crio-5ba9762f7d15703ed4a13e0c9e32f564d7dbe5a8191bed2c71f318c2a5767ce5 WatchSource:0}: Error finding container 5ba9762f7d15703ed4a13e0c9e32f564d7dbe5a8191bed2c71f318c2a5767ce5: Status 404 returned error can't find the container with id 5ba9762f7d15703ed4a13e0c9e32f564d7dbe5a8191bed2c71f318c2a5767ce5 Jul 13 14:53:20.836980 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:20.836961 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 14:53:21.132726 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:21.132696 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:21.666362 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:21.666316 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" event={"ID":"ec9ce717-43b0-47a4-85d0-6348d24a7dd2","Type":"ContainerStarted","Data":"b33c34bf789564b032bd6b2f26e124e7f642eac7283cb0d1fe3090ccf52ae93b"} Jul 13 14:53:21.666362 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:21.666368 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" event={"ID":"ec9ce717-43b0-47a4-85d0-6348d24a7dd2","Type":"ContainerStarted","Data":"5ba9762f7d15703ed4a13e0c9e32f564d7dbe5a8191bed2c71f318c2a5767ce5"} Jul 13 14:53:21.666877 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:21.666525 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" Jul 13 14:53:21.697357 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:21.697269 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" podStartSLOduration=2.089007688 podStartE2EDuration="2.697255113s" totalImagesPullingTime="608.247425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:21.694796523 +0000 UTC m=+1943.379216820" watchObservedRunningTime="2026-07-13 14:53:21.697255113 +0000 UTC m=+1943.381675420" Jul 13 14:53:24.362812 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.362766 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w"] Jul 13 14:53:24.363257 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.363014 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" podUID="ec9ce717-43b0-47a4-85d0-6348d24a7dd2" containerName="maas-api" containerID="cri-o://b33c34bf789564b032bd6b2f26e124e7f642eac7283cb0d1fe3090ccf52ae93b" gracePeriod=30 Jul 13 14:53:24.387411 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.387384 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:24.461503 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.461466 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:24.615571 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.615513 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" Jul 13 14:53:24.683106 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.683069 2566 generic.go:356] "Generic (PLEG): container finished" podID="ec9ce717-43b0-47a4-85d0-6348d24a7dd2" containerID="b33c34bf789564b032bd6b2f26e124e7f642eac7283cb0d1fe3090ccf52ae93b" exitCode=0 Jul 13 14:53:24.683283 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.683112 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" Jul 13 14:53:24.683283 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.683163 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" event={"ID":"ec9ce717-43b0-47a4-85d0-6348d24a7dd2","Type":"ContainerDied","Data":"b33c34bf789564b032bd6b2f26e124e7f642eac7283cb0d1fe3090ccf52ae93b"} Jul 13 14:53:24.683283 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.683226 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w" event={"ID":"ec9ce717-43b0-47a4-85d0-6348d24a7dd2","Type":"ContainerDied","Data":"5ba9762f7d15703ed4a13e0c9e32f564d7dbe5a8191bed2c71f318c2a5767ce5"} Jul 13 14:53:24.683283 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.683256 2566 scope.go:117] "RemoveContainer" containerID="b33c34bf789564b032bd6b2f26e124e7f642eac7283cb0d1fe3090ccf52ae93b" Jul 13 14:53:24.691891 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.691871 2566 scope.go:117] "RemoveContainer" containerID="b33c34bf789564b032bd6b2f26e124e7f642eac7283cb0d1fe3090ccf52ae93b" Jul 13 14:53:24.692154 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:53:24.692129 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33c34bf789564b032bd6b2f26e124e7f642eac7283cb0d1fe3090ccf52ae93b\": container with ID starting with b33c34bf789564b032bd6b2f26e124e7f642eac7283cb0d1fe3090ccf52ae93b not found: ID does not exist" containerID="b33c34bf789564b032bd6b2f26e124e7f642eac7283cb0d1fe3090ccf52ae93b" Jul 13 14:53:24.692253 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.692158 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33c34bf789564b032bd6b2f26e124e7f642eac7283cb0d1fe3090ccf52ae93b"} err="failed to get container status \"b33c34bf789564b032bd6b2f26e124e7f642eac7283cb0d1fe3090ccf52ae93b\": rpc error: code = NotFound desc = could not find container \"b33c34bf789564b032bd6b2f26e124e7f642eac7283cb0d1fe3090ccf52ae93b\": container with ID starting with b33c34bf789564b032bd6b2f26e124e7f642eac7283cb0d1fe3090ccf52ae93b not found: ID does not exist" Jul 13 14:53:24.720194 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.720170 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtsjq\" (UniqueName: \"kubernetes.io/projected/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-kube-api-access-dtsjq\") pod \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\" (UID: \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\") " Jul 13 14:53:24.720310 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.720252 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-maas-api-tls\") pod \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\" (UID: \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\") " Jul 13 14:53:24.720351 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.720313 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-openshift-service-ca\") pod \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\" (UID: \"ec9ce717-43b0-47a4-85d0-6348d24a7dd2\") " Jul 13 14:53:24.720657 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.720634 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec9ce717-43b0-47a4-85d0-6348d24a7dd2" (UID: "ec9ce717-43b0-47a4-85d0-6348d24a7dd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:53:24.722212 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.722190 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec9ce717-43b0-47a4-85d0-6348d24a7dd2" (UID: "ec9ce717-43b0-47a4-85d0-6348d24a7dd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:53:24.722296 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.722251 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-kube-api-access-dtsjq" (OuterVolumeSpecName: "kube-api-access-dtsjq") pod "ec9ce717-43b0-47a4-85d0-6348d24a7dd2" (UID: "ec9ce717-43b0-47a4-85d0-6348d24a7dd2"). InnerVolumeSpecName "kube-api-access-dtsjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:53:24.821480 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.821450 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtsjq\" (UniqueName: \"kubernetes.io/projected/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-kube-api-access-dtsjq\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:24.821480 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.821477 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-maas-api-tls\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:24.821480 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:24.821487 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec9ce717-43b0-47a4-85d0-6348d24a7dd2-openshift-service-ca\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:25.008141 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:25.008112 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w"] Jul 13 14:53:25.013337 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:25.013315 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d0ab5f95-6687f5c988-kcq5w"] Jul 13 14:53:25.447129 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:25.447088 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq"] Jul 13 14:53:25.447583 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:25.447325 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" podUID="01cbfaba-aba5-4eca-bea6-2f1ac084f144" containerName="istio-proxy" containerID="cri-o://35d687cf2d9d347cf0414b3da0fa831cc2441b541bbf984b2a40cca44b15701b" gracePeriod=30 Jul 13 14:53:25.487097 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:25.487063 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:26.751471 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:26.751430 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:26.873441 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:26.873399 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec9ce717-43b0-47a4-85d0-6348d24a7dd2" path="/var/lib/kubelet/pods/ec9ce717-43b0-47a4-85d0-6348d24a7dd2/volumes" Jul 13 14:53:28.398727 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:28.398693 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:30.704113 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.704086 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:30.715532 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.715456 2566 generic.go:356] "Generic (PLEG): container finished" podID="01cbfaba-aba5-4eca-bea6-2f1ac084f144" containerID="35d687cf2d9d347cf0414b3da0fa831cc2441b541bbf984b2a40cca44b15701b" exitCode=0 Jul 13 14:53:30.715532 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.715495 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" Jul 13 14:53:30.715678 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.715539 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" event={"ID":"01cbfaba-aba5-4eca-bea6-2f1ac084f144","Type":"ContainerDied","Data":"35d687cf2d9d347cf0414b3da0fa831cc2441b541bbf984b2a40cca44b15701b"} Jul 13 14:53:30.715678 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.715601 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq" event={"ID":"01cbfaba-aba5-4eca-bea6-2f1ac084f144","Type":"ContainerDied","Data":"a1b9784db598291f1704959df6a097f9800b349b5bf03d3c86a050efc39f4317"} Jul 13 14:53:30.715678 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.715623 2566 scope.go:117] "RemoveContainer" containerID="35d687cf2d9d347cf0414b3da0fa831cc2441b541bbf984b2a40cca44b15701b" Jul 13 14:53:30.724856 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.724840 2566 scope.go:117] "RemoveContainer" containerID="35d687cf2d9d347cf0414b3da0fa831cc2441b541bbf984b2a40cca44b15701b" Jul 13 14:53:30.725108 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:53:30.725085 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d687cf2d9d347cf0414b3da0fa831cc2441b541bbf984b2a40cca44b15701b\": container with ID starting with 35d687cf2d9d347cf0414b3da0fa831cc2441b541bbf984b2a40cca44b15701b not found: ID does not exist" containerID="35d687cf2d9d347cf0414b3da0fa831cc2441b541bbf984b2a40cca44b15701b" Jul 13 14:53:30.725174 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.725121 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d687cf2d9d347cf0414b3da0fa831cc2441b541bbf984b2a40cca44b15701b"} err="failed to get container status \"35d687cf2d9d347cf0414b3da0fa831cc2441b541bbf984b2a40cca44b15701b\": rpc error: code = NotFound desc = could not find container \"35d687cf2d9d347cf0414b3da0fa831cc2441b541bbf984b2a40cca44b15701b\": container with ID starting with 35d687cf2d9d347cf0414b3da0fa831cc2441b541bbf984b2a40cca44b15701b not found: ID does not exist" Jul 13 14:53:30.775097 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.775060 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgqpp\" (UniqueName: \"kubernetes.io/projected/01cbfaba-aba5-4eca-bea6-2f1ac084f144-kube-api-access-bgqpp\") pod \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " Jul 13 14:53:30.775097 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.775099 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-workload-socket\") pod \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " Jul 13 14:53:30.775314 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.775133 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-envoy\") pod \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " Jul 13 14:53:30.775314 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.775241 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-podinfo\") pod \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " Jul 13 14:53:30.775411 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.775393 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-credential-socket\") pod \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " Jul 13 14:53:30.775462 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.775409 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "01cbfaba-aba5-4eca-bea6-2f1ac084f144" (UID: "01cbfaba-aba5-4eca-bea6-2f1ac084f144"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:30.775462 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.775447 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istiod-ca-cert\") pod \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " Jul 13 14:53:30.775564 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.775486 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-data\") pod \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " Jul 13 14:53:30.775564 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.775518 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-workload-certs\") pod \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " Jul 13 14:53:30.775669 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.775587 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-token\") pod \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\" (UID: \"01cbfaba-aba5-4eca-bea6-2f1ac084f144\") " Jul 13 14:53:30.775719 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.775694 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "01cbfaba-aba5-4eca-bea6-2f1ac084f144" (UID: "01cbfaba-aba5-4eca-bea6-2f1ac084f144"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:30.775845 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.775806 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "01cbfaba-aba5-4eca-bea6-2f1ac084f144" (UID: "01cbfaba-aba5-4eca-bea6-2f1ac084f144"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:53:30.776083 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.775843 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "01cbfaba-aba5-4eca-bea6-2f1ac084f144" (UID: "01cbfaba-aba5-4eca-bea6-2f1ac084f144"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:30.776083 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.776002 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-workload-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:30.776083 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.776019 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-credential-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:30.776083 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.776029 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istiod-ca-cert\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:30.776083 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.776038 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-data" (OuterVolumeSpecName: "istio-data") pod "01cbfaba-aba5-4eca-bea6-2f1ac084f144" (UID: "01cbfaba-aba5-4eca-bea6-2f1ac084f144"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:30.776083 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.776044 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-workload-certs\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:30.777385 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.777333 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "01cbfaba-aba5-4eca-bea6-2f1ac084f144" (UID: "01cbfaba-aba5-4eca-bea6-2f1ac084f144"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:30.777482 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.777432 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01cbfaba-aba5-4eca-bea6-2f1ac084f144-kube-api-access-bgqpp" (OuterVolumeSpecName: "kube-api-access-bgqpp") pod "01cbfaba-aba5-4eca-bea6-2f1ac084f144" (UID: "01cbfaba-aba5-4eca-bea6-2f1ac084f144"). InnerVolumeSpecName "kube-api-access-bgqpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:53:30.777529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.777484 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "01cbfaba-aba5-4eca-bea6-2f1ac084f144" (UID: "01cbfaba-aba5-4eca-bea6-2f1ac084f144"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 14:53:30.777691 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.777672 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-token" (OuterVolumeSpecName: "istio-token") pod "01cbfaba-aba5-4eca-bea6-2f1ac084f144" (UID: "01cbfaba-aba5-4eca-bea6-2f1ac084f144"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:53:30.876683 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.876658 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-data\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:30.876683 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.876682 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-token\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:30.876854 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.876692 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgqpp\" (UniqueName: \"kubernetes.io/projected/01cbfaba-aba5-4eca-bea6-2f1ac084f144-kube-api-access-bgqpp\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:30.876854 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.876701 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-envoy\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:30.876854 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:30.876710 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01cbfaba-aba5-4eca-bea6-2f1ac084f144-istio-podinfo\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:31.033853 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:31.033766 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq"] Jul 13 14:53:31.040455 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:31.040429 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d0ab5f95-openshift-default-6c5d69b6c5-svcxq"] Jul 13 14:53:32.172246 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.172215 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:32.234903 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.234874 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:32.361752 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.361720 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh"] Jul 13 14:53:32.362230 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.362212 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec9ce717-43b0-47a4-85d0-6348d24a7dd2" containerName="maas-api" Jul 13 14:53:32.362230 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.362231 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec9ce717-43b0-47a4-85d0-6348d24a7dd2" containerName="maas-api" Jul 13 14:53:32.362375 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.362247 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01cbfaba-aba5-4eca-bea6-2f1ac084f144" containerName="istio-proxy" Jul 13 14:53:32.362375 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.362256 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="01cbfaba-aba5-4eca-bea6-2f1ac084f144" containerName="istio-proxy" Jul 13 14:53:32.362375 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.362320 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec9ce717-43b0-47a4-85d0-6348d24a7dd2" containerName="maas-api" Jul 13 14:53:32.362375 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.362327 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="01cbfaba-aba5-4eca-bea6-2f1ac084f144" containerName="istio-proxy" Jul 13 14:53:32.373073 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.373050 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.376054 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.376033 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a7c57961-openshift-default-dockercfg-dhzjt\"" Jul 13 14:53:32.377584 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.377562 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh"] Jul 13 14:53:32.398830 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.398798 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:32.491150 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.491061 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-podinfo\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.491150 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.491098 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-workload-socket\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.491150 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.491115 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-credential-socket\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.491369 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.491199 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-workload-certs\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.491369 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.491272 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-token\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.491369 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.491299 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbba4412-8bba-4d2d-b403-0f3424be8c70-istiod-ca-cert\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.491369 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.491316 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cbp4\" (UniqueName: \"kubernetes.io/projected/bbba4412-8bba-4d2d-b403-0f3424be8c70-kube-api-access-7cbp4\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.491369 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.491349 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-data\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.491525 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.491389 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-envoy\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.494450 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.494418 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:32.592341 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.592308 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-envoy\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.592341 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.592348 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-podinfo\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.592559 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.592370 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-workload-socket\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.592559 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.592392 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-credential-socket\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.592559 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.592429 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-workload-certs\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.592559 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.592497 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-token\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.592559 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.592530 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbba4412-8bba-4d2d-b403-0f3424be8c70-istiod-ca-cert\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.592559 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.592555 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cbp4\" (UniqueName: \"kubernetes.io/projected/bbba4412-8bba-4d2d-b403-0f3424be8c70-kube-api-access-7cbp4\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.592910 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.592590 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-data\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.592910 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.592841 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-workload-socket\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.592910 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.592876 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-credential-socket\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.593057 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.592924 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-workload-certs\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.593057 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.592965 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-data\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.593379 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.593359 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbba4412-8bba-4d2d-b403-0f3424be8c70-istiod-ca-cert\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.594684 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.594657 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-envoy\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.594839 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.594824 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-podinfo\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.602028 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.602008 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-token\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.602326 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.602304 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cbp4\" (UniqueName: \"kubernetes.io/projected/bbba4412-8bba-4d2d-b403-0f3424be8c70-kube-api-access-7cbp4\") pod \"e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.686850 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.686822 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:32.827623 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.827592 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh"] Jul 13 14:53:32.828840 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:53:32.828780 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbba4412_8bba_4d2d_b403_0f3424be8c70.slice/crio-a85a7d0f40f63377ea2342f84572c47e98b76ba9861963b17bab8ef138a366f7 WatchSource:0}: Error finding container a85a7d0f40f63377ea2342f84572c47e98b76ba9861963b17bab8ef138a366f7: Status 404 returned error can't find the container with id a85a7d0f40f63377ea2342f84572c47e98b76ba9861963b17bab8ef138a366f7 Jul 13 14:53:32.831299 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.831267 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:53:32.831407 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.831335 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:53:32.831407 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.831368 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:53:32.874562 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:32.874535 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01cbfaba-aba5-4eca-bea6-2f1ac084f144" path="/var/lib/kubelet/pods/01cbfaba-aba5-4eca-bea6-2f1ac084f144/volumes" Jul 13 14:53:33.510618 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.510585 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:33.606931 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.606902 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:33.683023 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.682992 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg"] Jul 13 14:53:33.686795 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.686759 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" Jul 13 14:53:33.689442 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.689415 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a7c57961-serving-cert\"" Jul 13 14:53:33.689593 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.689449 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rxm62\"" Jul 13 14:53:33.697813 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.697768 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg"] Jul 13 14:53:33.726662 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.726635 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:33.734979 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.734942 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" event={"ID":"bbba4412-8bba-4d2d-b403-0f3424be8c70","Type":"ContainerStarted","Data":"1835b481de7c17052f19473d36980254922dd8099cf36250424ec09da38e6884"} Jul 13 14:53:33.735126 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.735005 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" event={"ID":"bbba4412-8bba-4d2d-b403-0f3424be8c70","Type":"ContainerStarted","Data":"a85a7d0f40f63377ea2342f84572c47e98b76ba9861963b17bab8ef138a366f7"} Jul 13 14:53:33.756143 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.756100 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" podStartSLOduration=1.756084874 podStartE2EDuration="1.756084874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:33.755297105 +0000 UTC m=+1955.439717410" watchObservedRunningTime="2026-07-13 14:53:33.756084874 +0000 UTC m=+1955.440505179" Jul 13 14:53:33.803682 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.803609 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f96db4a-8078-4970-96e6-bc4e5989191f-maas-api-tls\") pod \"maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg\" (UID: \"4f96db4a-8078-4970-96e6-bc4e5989191f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" Jul 13 14:53:33.803845 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.803690 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmlfc\" (UniqueName: \"kubernetes.io/projected/4f96db4a-8078-4970-96e6-bc4e5989191f-kube-api-access-fmlfc\") pod \"maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg\" (UID: \"4f96db4a-8078-4970-96e6-bc4e5989191f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" Jul 13 14:53:33.803845 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.803770 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f96db4a-8078-4970-96e6-bc4e5989191f-openshift-service-ca\") pod \"maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg\" (UID: \"4f96db4a-8078-4970-96e6-bc4e5989191f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" Jul 13 14:53:33.904647 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.904614 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f96db4a-8078-4970-96e6-bc4e5989191f-openshift-service-ca\") pod \"maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg\" (UID: \"4f96db4a-8078-4970-96e6-bc4e5989191f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" Jul 13 14:53:33.904647 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.904652 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f96db4a-8078-4970-96e6-bc4e5989191f-maas-api-tls\") pod \"maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg\" (UID: \"4f96db4a-8078-4970-96e6-bc4e5989191f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" Jul 13 14:53:33.904901 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.904731 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmlfc\" (UniqueName: \"kubernetes.io/projected/4f96db4a-8078-4970-96e6-bc4e5989191f-kube-api-access-fmlfc\") pod \"maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg\" (UID: \"4f96db4a-8078-4970-96e6-bc4e5989191f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" Jul 13 14:53:33.905284 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.905267 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f96db4a-8078-4970-96e6-bc4e5989191f-openshift-service-ca\") pod \"maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg\" (UID: \"4f96db4a-8078-4970-96e6-bc4e5989191f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" Jul 13 14:53:33.907182 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.907158 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f96db4a-8078-4970-96e6-bc4e5989191f-maas-api-tls\") pod \"maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg\" (UID: \"4f96db4a-8078-4970-96e6-bc4e5989191f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" Jul 13 14:53:33.913308 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.913282 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmlfc\" (UniqueName: \"kubernetes.io/projected/4f96db4a-8078-4970-96e6-bc4e5989191f-kube-api-access-fmlfc\") pod \"maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg\" (UID: \"4f96db4a-8078-4970-96e6-bc4e5989191f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" Jul 13 14:53:33.938657 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:33.938622 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:34.000440 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:34.000414 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" Jul 13 14:53:34.144246 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:34.144224 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg"] Jul 13 14:53:34.145978 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:53:34.145952 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f96db4a_8078_4970_96e6_bc4e5989191f.slice/crio-443389a90499f6cf864c02117ea01114bf11dec785e10cfcda65700aeda10c0a WatchSource:0}: Error finding container 443389a90499f6cf864c02117ea01114bf11dec785e10cfcda65700aeda10c0a: Status 404 returned error can't find the container with id 443389a90499f6cf864c02117ea01114bf11dec785e10cfcda65700aeda10c0a Jul 13 14:53:34.445275 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:34.445183 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:34.545256 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:34.545219 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:34.687393 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:34.687362 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:34.692229 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:34.692201 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:34.743360 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:34.743274 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" event={"ID":"4f96db4a-8078-4970-96e6-bc4e5989191f","Type":"ContainerStarted","Data":"9e0d7fc3efa4407f5fe0aed7d9954f67ae43a503b0a35b7acef8f4ae4505fcc1"} Jul 13 14:53:34.743530 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:34.743354 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:34.743530 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:34.743400 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" event={"ID":"4f96db4a-8078-4970-96e6-bc4e5989191f","Type":"ContainerStarted","Data":"443389a90499f6cf864c02117ea01114bf11dec785e10cfcda65700aeda10c0a"} Jul 13 14:53:34.743650 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:34.743580 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" Jul 13 14:53:34.744343 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:34.744323 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:34.761138 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:34.761092 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" podStartSLOduration=1.333969119 podStartE2EDuration="1.761079708s" totalImagesPullingTime="427.110589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:53:34.759836497 +0000 UTC m=+1956.444256799" watchObservedRunningTime="2026-07-13 14:53:34.761079708 +0000 UTC m=+1956.445500049" Jul 13 14:53:35.774936 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:35.774902 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:37.922902 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:37.921352 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg"] Jul 13 14:53:37.922902 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:37.921613 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" podUID="4f96db4a-8078-4970-96e6-bc4e5989191f" containerName="maas-api" containerID="cri-o://9e0d7fc3efa4407f5fe0aed7d9954f67ae43a503b0a35b7acef8f4ae4505fcc1" gracePeriod=30 Jul 13 14:53:37.956642 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:37.956415 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:38.019231 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.019186 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:38.137039 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.137004 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh"] Jul 13 14:53:38.137232 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.137207 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" podUID="bbba4412-8bba-4d2d-b403-0f3424be8c70" containerName="istio-proxy" containerID="cri-o://1835b481de7c17052f19473d36980254922dd8099cf36250424ec09da38e6884" gracePeriod=30 Jul 13 14:53:38.159011 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.158973 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:38.199184 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.199158 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" Jul 13 14:53:38.275042 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.274952 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:38.344996 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.344966 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f96db4a-8078-4970-96e6-bc4e5989191f-openshift-service-ca\") pod \"4f96db4a-8078-4970-96e6-bc4e5989191f\" (UID: \"4f96db4a-8078-4970-96e6-bc4e5989191f\") " Jul 13 14:53:38.344996 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.345007 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f96db4a-8078-4970-96e6-bc4e5989191f-maas-api-tls\") pod \"4f96db4a-8078-4970-96e6-bc4e5989191f\" (UID: \"4f96db4a-8078-4970-96e6-bc4e5989191f\") " Jul 13 14:53:38.345191 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.345068 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmlfc\" (UniqueName: \"kubernetes.io/projected/4f96db4a-8078-4970-96e6-bc4e5989191f-kube-api-access-fmlfc\") pod \"4f96db4a-8078-4970-96e6-bc4e5989191f\" (UID: \"4f96db4a-8078-4970-96e6-bc4e5989191f\") " Jul 13 14:53:38.345314 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.345293 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f96db4a-8078-4970-96e6-bc4e5989191f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4f96db4a-8078-4970-96e6-bc4e5989191f" (UID: "4f96db4a-8078-4970-96e6-bc4e5989191f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:53:38.347221 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.347199 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f96db4a-8078-4970-96e6-bc4e5989191f-kube-api-access-fmlfc" (OuterVolumeSpecName: "kube-api-access-fmlfc") pod "4f96db4a-8078-4970-96e6-bc4e5989191f" (UID: "4f96db4a-8078-4970-96e6-bc4e5989191f"). InnerVolumeSpecName "kube-api-access-fmlfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:53:38.347315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.347196 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f96db4a-8078-4970-96e6-bc4e5989191f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4f96db4a-8078-4970-96e6-bc4e5989191f" (UID: "4f96db4a-8078-4970-96e6-bc4e5989191f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:53:38.446094 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.446013 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f96db4a-8078-4970-96e6-bc4e5989191f-openshift-service-ca\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:38.446094 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.446042 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f96db4a-8078-4970-96e6-bc4e5989191f-maas-api-tls\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:38.446094 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.446052 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmlfc\" (UniqueName: \"kubernetes.io/projected/4f96db4a-8078-4970-96e6-bc4e5989191f-kube-api-access-fmlfc\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:38.768943 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.768862 2566 generic.go:356] "Generic (PLEG): container finished" podID="4f96db4a-8078-4970-96e6-bc4e5989191f" containerID="9e0d7fc3efa4407f5fe0aed7d9954f67ae43a503b0a35b7acef8f4ae4505fcc1" exitCode=0 Jul 13 14:53:38.768943 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.768907 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" Jul 13 14:53:38.769133 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.768954 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" event={"ID":"4f96db4a-8078-4970-96e6-bc4e5989191f","Type":"ContainerDied","Data":"9e0d7fc3efa4407f5fe0aed7d9954f67ae43a503b0a35b7acef8f4ae4505fcc1"} Jul 13 14:53:38.769133 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.769002 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg" event={"ID":"4f96db4a-8078-4970-96e6-bc4e5989191f","Type":"ContainerDied","Data":"443389a90499f6cf864c02117ea01114bf11dec785e10cfcda65700aeda10c0a"} Jul 13 14:53:38.769133 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.769022 2566 scope.go:117] "RemoveContainer" containerID="9e0d7fc3efa4407f5fe0aed7d9954f67ae43a503b0a35b7acef8f4ae4505fcc1" Jul 13 14:53:38.780865 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.780848 2566 scope.go:117] "RemoveContainer" containerID="9e0d7fc3efa4407f5fe0aed7d9954f67ae43a503b0a35b7acef8f4ae4505fcc1" Jul 13 14:53:38.781158 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:53:38.781132 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0d7fc3efa4407f5fe0aed7d9954f67ae43a503b0a35b7acef8f4ae4505fcc1\": container with ID starting with 9e0d7fc3efa4407f5fe0aed7d9954f67ae43a503b0a35b7acef8f4ae4505fcc1 not found: ID does not exist" containerID="9e0d7fc3efa4407f5fe0aed7d9954f67ae43a503b0a35b7acef8f4ae4505fcc1" Jul 13 14:53:38.781231 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.781168 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0d7fc3efa4407f5fe0aed7d9954f67ae43a503b0a35b7acef8f4ae4505fcc1"} err="failed to get container status \"9e0d7fc3efa4407f5fe0aed7d9954f67ae43a503b0a35b7acef8f4ae4505fcc1\": rpc error: code = NotFound desc = could not find container \"9e0d7fc3efa4407f5fe0aed7d9954f67ae43a503b0a35b7acef8f4ae4505fcc1\": container with ID starting with 9e0d7fc3efa4407f5fe0aed7d9954f67ae43a503b0a35b7acef8f4ae4505fcc1 not found: ID does not exist" Jul 13 14:53:38.794892 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.794868 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg"] Jul 13 14:53:38.799873 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.799845 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7c57961-6dbbb65f78-k5rfg"] Jul 13 14:53:38.867193 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:38.867161 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f96db4a-8078-4970-96e6-bc4e5989191f" path="/var/lib/kubelet/pods/4f96db4a-8078-4970-96e6-bc4e5989191f/volumes" Jul 13 14:53:39.570851 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:39.570815 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:43.383486 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.383460 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:43.492311 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.492230 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-token\") pod \"bbba4412-8bba-4d2d-b403-0f3424be8c70\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " Jul 13 14:53:43.492311 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.492270 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-podinfo\") pod \"bbba4412-8bba-4d2d-b403-0f3424be8c70\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " Jul 13 14:53:43.492485 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.492313 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-credential-socket\") pod \"bbba4412-8bba-4d2d-b403-0f3424be8c70\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " Jul 13 14:53:43.492485 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.492473 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-workload-socket\") pod \"bbba4412-8bba-4d2d-b403-0f3424be8c70\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " Jul 13 14:53:43.492562 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.492502 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cbp4\" (UniqueName: \"kubernetes.io/projected/bbba4412-8bba-4d2d-b403-0f3424be8c70-kube-api-access-7cbp4\") pod \"bbba4412-8bba-4d2d-b403-0f3424be8c70\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " Jul 13 14:53:43.492562 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.492525 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-envoy\") pod \"bbba4412-8bba-4d2d-b403-0f3424be8c70\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " Jul 13 14:53:43.492562 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.492558 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbba4412-8bba-4d2d-b403-0f3424be8c70-istiod-ca-cert\") pod \"bbba4412-8bba-4d2d-b403-0f3424be8c70\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " Jul 13 14:53:43.492757 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.492603 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-workload-certs\") pod \"bbba4412-8bba-4d2d-b403-0f3424be8c70\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " Jul 13 14:53:43.492757 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.492556 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bbba4412-8bba-4d2d-b403-0f3424be8c70" (UID: "bbba4412-8bba-4d2d-b403-0f3424be8c70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:43.492757 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.492639 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-data\") pod \"bbba4412-8bba-4d2d-b403-0f3424be8c70\" (UID: \"bbba4412-8bba-4d2d-b403-0f3424be8c70\") " Jul 13 14:53:43.492961 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.492744 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bbba4412-8bba-4d2d-b403-0f3424be8c70" (UID: "bbba4412-8bba-4d2d-b403-0f3424be8c70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:43.492961 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.492844 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bbba4412-8bba-4d2d-b403-0f3424be8c70" (UID: "bbba4412-8bba-4d2d-b403-0f3424be8c70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:43.492961 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.492893 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbba4412-8bba-4d2d-b403-0f3424be8c70-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bbba4412-8bba-4d2d-b403-0f3424be8c70" (UID: "bbba4412-8bba-4d2d-b403-0f3424be8c70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:53:43.493132 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.493061 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-credential-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:43.493132 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.493081 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-workload-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:43.493132 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.493097 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbba4412-8bba-4d2d-b403-0f3424be8c70-istiod-ca-cert\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:43.493132 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.493112 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-workload-certs\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:43.493811 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.493756 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-data" (OuterVolumeSpecName: "istio-data") pod "bbba4412-8bba-4d2d-b403-0f3424be8c70" (UID: "bbba4412-8bba-4d2d-b403-0f3424be8c70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:43.494613 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.494586 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bbba4412-8bba-4d2d-b403-0f3424be8c70" (UID: "bbba4412-8bba-4d2d-b403-0f3424be8c70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 14:53:43.494721 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.494692 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bbba4412-8bba-4d2d-b403-0f3424be8c70" (UID: "bbba4412-8bba-4d2d-b403-0f3424be8c70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:53:43.494721 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.494704 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbba4412-8bba-4d2d-b403-0f3424be8c70-kube-api-access-7cbp4" (OuterVolumeSpecName: "kube-api-access-7cbp4") pod "bbba4412-8bba-4d2d-b403-0f3424be8c70" (UID: "bbba4412-8bba-4d2d-b403-0f3424be8c70"). InnerVolumeSpecName "kube-api-access-7cbp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:53:43.494848 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.494738 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-token" (OuterVolumeSpecName: "istio-token") pod "bbba4412-8bba-4d2d-b403-0f3424be8c70" (UID: "bbba4412-8bba-4d2d-b403-0f3424be8c70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:53:43.593969 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.593928 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-token\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:43.593969 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.593964 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-podinfo\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:43.593969 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.593974 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cbp4\" (UniqueName: \"kubernetes.io/projected/bbba4412-8bba-4d2d-b403-0f3424be8c70-kube-api-access-7cbp4\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:43.593969 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.593984 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-envoy\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:43.594205 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.593992 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbba4412-8bba-4d2d-b403-0f3424be8c70-istio-data\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:53:43.796663 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.796577 2566 generic.go:356] "Generic (PLEG): container finished" podID="bbba4412-8bba-4d2d-b403-0f3424be8c70" containerID="1835b481de7c17052f19473d36980254922dd8099cf36250424ec09da38e6884" exitCode=0 Jul 13 14:53:43.796663 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.796619 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" Jul 13 14:53:43.796875 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.796656 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" event={"ID":"bbba4412-8bba-4d2d-b403-0f3424be8c70","Type":"ContainerDied","Data":"1835b481de7c17052f19473d36980254922dd8099cf36250424ec09da38e6884"} Jul 13 14:53:43.796875 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.796707 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh" event={"ID":"bbba4412-8bba-4d2d-b403-0f3424be8c70","Type":"ContainerDied","Data":"a85a7d0f40f63377ea2342f84572c47e98b76ba9861963b17bab8ef138a366f7"} Jul 13 14:53:43.796875 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.796729 2566 scope.go:117] "RemoveContainer" containerID="1835b481de7c17052f19473d36980254922dd8099cf36250424ec09da38e6884" Jul 13 14:53:43.806180 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.806162 2566 scope.go:117] "RemoveContainer" containerID="1835b481de7c17052f19473d36980254922dd8099cf36250424ec09da38e6884" Jul 13 14:53:43.806437 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:53:43.806419 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1835b481de7c17052f19473d36980254922dd8099cf36250424ec09da38e6884\": container with ID starting with 1835b481de7c17052f19473d36980254922dd8099cf36250424ec09da38e6884 not found: ID does not exist" containerID="1835b481de7c17052f19473d36980254922dd8099cf36250424ec09da38e6884" Jul 13 14:53:43.806483 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.806447 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1835b481de7c17052f19473d36980254922dd8099cf36250424ec09da38e6884"} err="failed to get container status \"1835b481de7c17052f19473d36980254922dd8099cf36250424ec09da38e6884\": rpc error: code = NotFound desc = could not find container \"1835b481de7c17052f19473d36980254922dd8099cf36250424ec09da38e6884\": container with ID starting with 1835b481de7c17052f19473d36980254922dd8099cf36250424ec09da38e6884 not found: ID does not exist" Jul 13 14:53:43.821935 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.821914 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh"] Jul 13 14:53:43.828991 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:43.828970 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a7c57961-openshift-default-7888654b87-xm6dh"] Jul 13 14:53:44.867617 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:44.867583 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbba4412-8bba-4d2d-b403-0f3424be8c70" path="/var/lib/kubelet/pods/bbba4412-8bba-4d2d-b403-0f3424be8c70/volumes" Jul 13 14:53:45.217166 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:45.217075 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:45.284742 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:45.284711 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:45.721863 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:45.721833 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:47.365078 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:47.365041 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:54.085713 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:54.085638 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:54.192754 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:54.192673 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:54.296507 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:54.296475 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:54.872413 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:54.872380 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:53:54.971769 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:53:54.971740 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:20.892933 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:54:20.892892 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:27.417049 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:54:27.417013 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:27.475229 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:54:27.475201 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:27.604034 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:54:27.604002 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:27.701818 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:54:27.701705 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:43.818251 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:54:43.818211 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:50.354560 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:54:50.354522 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:50.468623 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:54:50.468576 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:54:50.569366 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:54:50.569331 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:01.077774 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:01.077737 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:01.881313 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:01.881280 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:08.515882 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:08.515815 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:08.593590 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:08.593555 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:08.678986 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:08.678953 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:08.780308 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:08.780222 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:09.508046 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:09.508011 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:09.670774 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:09.670731 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:09.835615 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:09.835575 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:11.455194 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:11.455153 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:17.678705 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:17.678673 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:17.778705 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:17.778668 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:18.379128 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:18.379094 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:24.742962 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:24.742870 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:25.294638 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.294600 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc"] Jul 13 14:55:25.295014 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.295001 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f96db4a-8078-4970-96e6-bc4e5989191f" containerName="maas-api" Jul 13 14:55:25.295014 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.295016 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f96db4a-8078-4970-96e6-bc4e5989191f" containerName="maas-api" Jul 13 14:55:25.295161 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.295031 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbba4412-8bba-4d2d-b403-0f3424be8c70" containerName="istio-proxy" Jul 13 14:55:25.295161 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.295038 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbba4412-8bba-4d2d-b403-0f3424be8c70" containerName="istio-proxy" Jul 13 14:55:25.295161 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.295107 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbba4412-8bba-4d2d-b403-0f3424be8c70" containerName="istio-proxy" Jul 13 14:55:25.295161 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.295121 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f96db4a-8078-4970-96e6-bc4e5989191f" containerName="maas-api" Jul 13 14:55:25.298119 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.298102 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.301210 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.301190 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3cced6c6-openshift-default-dockercfg-jw4pf\"" Jul 13 14:55:25.321081 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.321054 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc"] Jul 13 14:55:25.457993 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.457960 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:25.475906 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.475880 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-envoy\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.476061 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.475924 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-data\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.476061 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.475956 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-workload-certs\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.476061 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.476022 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeef5ae0-8dc9-452e-980c-56d673e0d857-istiod-ca-cert\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.476170 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.476065 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-credential-socket\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.476170 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.476084 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-podinfo\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.476170 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.476110 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-token\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.476170 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.476149 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-workload-socket\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.476299 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.476222 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655vv\" (UniqueName: \"kubernetes.io/projected/eeef5ae0-8dc9-452e-980c-56d673e0d857-kube-api-access-655vv\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.529296 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.529256 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:25.577187 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.577154 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-credential-socket\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.577187 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.577189 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-podinfo\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.577416 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.577206 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-token\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.577416 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.577224 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-workload-socket\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.577416 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.577252 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655vv\" (UniqueName: \"kubernetes.io/projected/eeef5ae0-8dc9-452e-980c-56d673e0d857-kube-api-access-655vv\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.577416 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.577306 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-envoy\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.577416 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.577335 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-data\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.577416 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.577360 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-workload-certs\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.577416 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.577398 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeef5ae0-8dc9-452e-980c-56d673e0d857-istiod-ca-cert\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.577767 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.577631 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-credential-socket\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.577767 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.577656 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-workload-socket\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.577767 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.577701 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-data\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.577903 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.577886 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-workload-certs\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.578055 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.578039 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeef5ae0-8dc9-452e-980c-56d673e0d857-istiod-ca-cert\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.579743 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.579720 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-envoy\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.579852 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.579835 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-podinfo\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.585835 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.585814 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-655vv\" (UniqueName: \"kubernetes.io/projected/eeef5ae0-8dc9-452e-980c-56d673e0d857-kube-api-access-655vv\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.585986 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.585967 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-token\") pod \"e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.608316 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.608292 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:25.750965 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.750935 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc"] Jul 13 14:55:25.752325 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:55:25.752297 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeef5ae0_8dc9_452e_980c_56d673e0d857.slice/crio-6a56f2902b133dfdebc5db37f3b433e31272c9e923aa6fb4016be7eb2b2ced1d WatchSource:0}: Error finding container 6a56f2902b133dfdebc5db37f3b433e31272c9e923aa6fb4016be7eb2b2ced1d: Status 404 returned error can't find the container with id 6a56f2902b133dfdebc5db37f3b433e31272c9e923aa6fb4016be7eb2b2ced1d Jul 13 14:55:25.754559 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.754529 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:55:25.754662 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.754602 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:55:25.754662 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:25.754646 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:55:26.346816 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:26.346767 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" event={"ID":"eeef5ae0-8dc9-452e-980c-56d673e0d857","Type":"ContainerStarted","Data":"de8774b83dd9ee38eb846dfbc1c2d8e9f55ad125621eccbd4a61e5085e7c2ee9"} Jul 13 14:55:26.347023 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:26.346836 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" event={"ID":"eeef5ae0-8dc9-452e-980c-56d673e0d857","Type":"ContainerStarted","Data":"6a56f2902b133dfdebc5db37f3b433e31272c9e923aa6fb4016be7eb2b2ced1d"} Jul 13 14:55:26.370539 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:26.370491 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" podStartSLOduration=1.3704760999999999 podStartE2EDuration="1.3704761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:55:26.36775506 +0000 UTC m=+2068.052175367" watchObservedRunningTime="2026-07-13 14:55:26.3704761 +0000 UTC m=+2068.054896405" Jul 13 14:55:26.609290 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:26.609211 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:26.614100 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:26.614073 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:27.352195 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:27.352160 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:27.353024 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:27.353003 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:28.308035 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.308007 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:28.408702 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.408667 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:28.602833 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.602800 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm"] Jul 13 14:55:28.606498 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.606474 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" Jul 13 14:55:28.609627 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.609604 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3cced6c6-serving-cert\"" Jul 13 14:55:28.609848 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.609834 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rxm62\"" Jul 13 14:55:28.618684 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.618664 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm"] Jul 13 14:55:28.636665 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.636636 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:28.706346 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.706314 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4aa2b1d-0326-4177-8a37-1d5e8558b725-maas-api-tls\") pod \"maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm\" (UID: \"e4aa2b1d-0326-4177-8a37-1d5e8558b725\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" Jul 13 14:55:28.706529 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.706456 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnd7g\" (UniqueName: \"kubernetes.io/projected/e4aa2b1d-0326-4177-8a37-1d5e8558b725-kube-api-access-hnd7g\") pod \"maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm\" (UID: \"e4aa2b1d-0326-4177-8a37-1d5e8558b725\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" Jul 13 14:55:28.706583 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.706556 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4aa2b1d-0326-4177-8a37-1d5e8558b725-openshift-service-ca\") pod \"maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm\" (UID: \"e4aa2b1d-0326-4177-8a37-1d5e8558b725\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" Jul 13 14:55:28.757209 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.757176 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:28.807464 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.807431 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4aa2b1d-0326-4177-8a37-1d5e8558b725-maas-api-tls\") pod \"maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm\" (UID: \"e4aa2b1d-0326-4177-8a37-1d5e8558b725\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" Jul 13 14:55:28.807638 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.807605 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnd7g\" (UniqueName: \"kubernetes.io/projected/e4aa2b1d-0326-4177-8a37-1d5e8558b725-kube-api-access-hnd7g\") pod \"maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm\" (UID: \"e4aa2b1d-0326-4177-8a37-1d5e8558b725\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" Jul 13 14:55:28.807718 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.807702 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4aa2b1d-0326-4177-8a37-1d5e8558b725-openshift-service-ca\") pod \"maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm\" (UID: \"e4aa2b1d-0326-4177-8a37-1d5e8558b725\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" Jul 13 14:55:28.808313 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.808289 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4aa2b1d-0326-4177-8a37-1d5e8558b725-openshift-service-ca\") pod \"maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm\" (UID: \"e4aa2b1d-0326-4177-8a37-1d5e8558b725\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" Jul 13 14:55:28.809893 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.809873 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4aa2b1d-0326-4177-8a37-1d5e8558b725-maas-api-tls\") pod \"maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm\" (UID: \"e4aa2b1d-0326-4177-8a37-1d5e8558b725\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" Jul 13 14:55:28.820227 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.820205 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnd7g\" (UniqueName: \"kubernetes.io/projected/e4aa2b1d-0326-4177-8a37-1d5e8558b725-kube-api-access-hnd7g\") pod \"maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm\" (UID: \"e4aa2b1d-0326-4177-8a37-1d5e8558b725\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" Jul 13 14:55:28.917692 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:28.917602 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" Jul 13 14:55:29.067274 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:29.067246 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm"] Jul 13 14:55:29.068559 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:55:29.068531 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4aa2b1d_0326_4177_8a37_1d5e8558b725.slice/crio-10ecc72ea5252a84ef6faae340e8399929c4642195719e2eb357921fd9cafae5 WatchSource:0}: Error finding container 10ecc72ea5252a84ef6faae340e8399929c4642195719e2eb357921fd9cafae5: Status 404 returned error can't find the container with id 10ecc72ea5252a84ef6faae340e8399929c4642195719e2eb357921fd9cafae5 Jul 13 14:55:29.132121 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:29.132086 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:29.232991 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:29.232878 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:29.364000 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:29.363969 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" event={"ID":"e4aa2b1d-0326-4177-8a37-1d5e8558b725","Type":"ContainerStarted","Data":"10ecc72ea5252a84ef6faae340e8399929c4642195719e2eb357921fd9cafae5"} Jul 13 14:55:29.850195 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:29.850156 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:30.371534 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:30.371502 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" event={"ID":"e4aa2b1d-0326-4177-8a37-1d5e8558b725","Type":"ContainerStarted","Data":"e26742ccfa873a2a57b3d5a19e8b16835ca872c0987ca73592235a5467964753"} Jul 13 14:55:30.371739 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:30.371614 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" Jul 13 14:55:30.393122 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:30.393082 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" podStartSLOduration=1.781074745 podStartE2EDuration="2.393069368s" totalImagesPullingTime="611.994623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:55:30.391024273 +0000 UTC m=+2072.075444578" watchObservedRunningTime="2026-07-13 14:55:30.393069368 +0000 UTC m=+2072.077489673" Jul 13 14:55:33.026114 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:33.026066 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:33.471327 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:33.471295 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:33.570852 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:33.570815 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:33.917151 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:33.917110 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:36.382722 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:36.382694 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" Jul 13 14:55:40.597425 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:40.597386 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:40.737103 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:40.737062 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:41.036627 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:41.036540 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:41.470770 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:41.470738 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm"] Jul 13 14:55:41.471263 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:41.471226 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" podUID="e4aa2b1d-0326-4177-8a37-1d5e8558b725" containerName="maas-api" containerID="cri-o://e26742ccfa873a2a57b3d5a19e8b16835ca872c0987ca73592235a5467964753" gracePeriod=30 Jul 13 14:55:41.486693 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:41.486515 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:41.585928 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:41.585895 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:41.724766 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:41.724704 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" Jul 13 14:55:41.818635 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:41.818608 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4aa2b1d-0326-4177-8a37-1d5e8558b725-maas-api-tls\") pod \"e4aa2b1d-0326-4177-8a37-1d5e8558b725\" (UID: \"e4aa2b1d-0326-4177-8a37-1d5e8558b725\") " Jul 13 14:55:41.819704 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:41.818643 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4aa2b1d-0326-4177-8a37-1d5e8558b725-openshift-service-ca\") pod \"e4aa2b1d-0326-4177-8a37-1d5e8558b725\" (UID: \"e4aa2b1d-0326-4177-8a37-1d5e8558b725\") " Jul 13 14:55:41.819704 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:41.818676 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnd7g\" (UniqueName: \"kubernetes.io/projected/e4aa2b1d-0326-4177-8a37-1d5e8558b725-kube-api-access-hnd7g\") pod \"e4aa2b1d-0326-4177-8a37-1d5e8558b725\" (UID: \"e4aa2b1d-0326-4177-8a37-1d5e8558b725\") " Jul 13 14:55:41.819704 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:41.819035 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4aa2b1d-0326-4177-8a37-1d5e8558b725-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4aa2b1d-0326-4177-8a37-1d5e8558b725" (UID: "e4aa2b1d-0326-4177-8a37-1d5e8558b725"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:55:41.821235 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:41.821210 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4aa2b1d-0326-4177-8a37-1d5e8558b725-kube-api-access-hnd7g" (OuterVolumeSpecName: "kube-api-access-hnd7g") pod "e4aa2b1d-0326-4177-8a37-1d5e8558b725" (UID: "e4aa2b1d-0326-4177-8a37-1d5e8558b725"). InnerVolumeSpecName "kube-api-access-hnd7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:55:41.821450 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:41.821428 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4aa2b1d-0326-4177-8a37-1d5e8558b725-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4aa2b1d-0326-4177-8a37-1d5e8558b725" (UID: "e4aa2b1d-0326-4177-8a37-1d5e8558b725"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:55:41.919494 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:41.919468 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4aa2b1d-0326-4177-8a37-1d5e8558b725-maas-api-tls\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:55:41.919494 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:41.919495 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4aa2b1d-0326-4177-8a37-1d5e8558b725-openshift-service-ca\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:55:41.919680 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:41.919505 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnd7g\" (UniqueName: \"kubernetes.io/projected/e4aa2b1d-0326-4177-8a37-1d5e8558b725-kube-api-access-hnd7g\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:55:42.354221 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.354183 2566 patch_prober.go:28] interesting pod/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 13 14:55:42.354395 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.354237 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" podUID="eeef5ae0-8dc9-452e-980c-56d673e0d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 13 14:55:42.438838 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.438780 2566 generic.go:356] "Generic (PLEG): container finished" podID="e4aa2b1d-0326-4177-8a37-1d5e8558b725" containerID="e26742ccfa873a2a57b3d5a19e8b16835ca872c0987ca73592235a5467964753" exitCode=0 Jul 13 14:55:42.439004 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.438851 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" Jul 13 14:55:42.439004 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.438883 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" event={"ID":"e4aa2b1d-0326-4177-8a37-1d5e8558b725","Type":"ContainerDied","Data":"e26742ccfa873a2a57b3d5a19e8b16835ca872c0987ca73592235a5467964753"} Jul 13 14:55:42.439004 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.438935 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm" event={"ID":"e4aa2b1d-0326-4177-8a37-1d5e8558b725","Type":"ContainerDied","Data":"10ecc72ea5252a84ef6faae340e8399929c4642195719e2eb357921fd9cafae5"} Jul 13 14:55:42.439004 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.438958 2566 scope.go:117] "RemoveContainer" containerID="e26742ccfa873a2a57b3d5a19e8b16835ca872c0987ca73592235a5467964753" Jul 13 14:55:42.440719 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.440697 2566 generic.go:356] "Generic (PLEG): container finished" podID="eeef5ae0-8dc9-452e-980c-56d673e0d857" containerID="de8774b83dd9ee38eb846dfbc1c2d8e9f55ad125621eccbd4a61e5085e7c2ee9" exitCode=0 Jul 13 14:55:42.440872 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.440738 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" event={"ID":"eeef5ae0-8dc9-452e-980c-56d673e0d857","Type":"ContainerDied","Data":"de8774b83dd9ee38eb846dfbc1c2d8e9f55ad125621eccbd4a61e5085e7c2ee9"} Jul 13 14:55:42.441107 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.441092 2566 scope.go:117] "RemoveContainer" containerID="de8774b83dd9ee38eb846dfbc1c2d8e9f55ad125621eccbd4a61e5085e7c2ee9" Jul 13 14:55:42.442246 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.442198 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:55:42.442339 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.442299 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:55:42.442380 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.442346 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:55:42.448891 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.448876 2566 scope.go:117] "RemoveContainer" containerID="e26742ccfa873a2a57b3d5a19e8b16835ca872c0987ca73592235a5467964753" Jul 13 14:55:42.449193 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:55:42.449174 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26742ccfa873a2a57b3d5a19e8b16835ca872c0987ca73592235a5467964753\": container with ID starting with e26742ccfa873a2a57b3d5a19e8b16835ca872c0987ca73592235a5467964753 not found: ID does not exist" containerID="e26742ccfa873a2a57b3d5a19e8b16835ca872c0987ca73592235a5467964753" Jul 13 14:55:42.449253 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.449200 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26742ccfa873a2a57b3d5a19e8b16835ca872c0987ca73592235a5467964753"} err="failed to get container status \"e26742ccfa873a2a57b3d5a19e8b16835ca872c0987ca73592235a5467964753\": rpc error: code = NotFound desc = could not find container \"e26742ccfa873a2a57b3d5a19e8b16835ca872c0987ca73592235a5467964753\": container with ID starting with e26742ccfa873a2a57b3d5a19e8b16835ca872c0987ca73592235a5467964753 not found: ID does not exist" Jul 13 14:55:42.480247 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.480219 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm"] Jul 13 14:55:42.488236 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.488210 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3cced6c6-7465c66d9b-lq4tm"] Jul 13 14:55:42.867575 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:42.867546 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4aa2b1d-0326-4177-8a37-1d5e8558b725" path="/var/lib/kubelet/pods/e4aa2b1d-0326-4177-8a37-1d5e8558b725/volumes" Jul 13 14:55:43.452530 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:43.452490 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" event={"ID":"eeef5ae0-8dc9-452e-980c-56d673e0d857","Type":"ContainerStarted","Data":"7ad847183de3eaf51522b084cb90c5bd1e2372e1d3df49fd43051eeef0071a79"} Jul 13 14:55:43.608613 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:43.608579 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:43.608852 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:43.608830 2566 patch_prober.go:28] interesting pod/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 13 14:55:43.608908 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:43.608879 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" podUID="eeef5ae0-8dc9-452e-980c-56d673e0d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 13 14:55:44.608802 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:44.608746 2566 patch_prober.go:28] interesting pod/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 13 14:55:44.609188 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:44.608822 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" podUID="eeef5ae0-8dc9-452e-980c-56d673e0d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 13 14:55:45.609654 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:45.609617 2566 patch_prober.go:28] interesting pod/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 13 14:55:45.610151 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:45.609691 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" podUID="eeef5ae0-8dc9-452e-980c-56d673e0d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 13 14:55:46.609085 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:46.609049 2566 patch_prober.go:28] interesting pod/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 13 14:55:46.609264 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:46.609126 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" podUID="eeef5ae0-8dc9-452e-980c-56d673e0d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 13 14:55:47.613076 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:47.613047 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:47.613459 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:47.613315 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:47.614066 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:47.614049 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:47.806052 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:47.806018 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:47.906933 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:47.906842 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:48.050668 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:48.050638 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc"] Jul 13 14:55:48.069237 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:48.069208 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:48.332069 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:48.332031 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:48.435852 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:48.435814 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:48.481512 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:48.481478 2566 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" secret="" err="secret \"e2e-mt-3cced6c6-openshift-default-dockercfg-jw4pf\" not found" Jul 13 14:55:48.533843 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:48.533808 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:48.661413 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:48.661327 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:49.486727 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:49.486684 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" podUID="eeef5ae0-8dc9-452e-980c-56d673e0d857" containerName="istio-proxy" containerID="cri-o://7ad847183de3eaf51522b084cb90c5bd1e2372e1d3df49fd43051eeef0071a79" gracePeriod=30 Jul 13 14:55:54.743865 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.743836 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:54.846378 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.846342 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-data\") pod \"eeef5ae0-8dc9-452e-980c-56d673e0d857\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " Jul 13 14:55:54.846576 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.846408 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-token\") pod \"eeef5ae0-8dc9-452e-980c-56d673e0d857\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " Jul 13 14:55:54.846576 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.846436 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-workload-socket\") pod \"eeef5ae0-8dc9-452e-980c-56d673e0d857\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " Jul 13 14:55:54.846576 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.846460 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-655vv\" (UniqueName: \"kubernetes.io/projected/eeef5ae0-8dc9-452e-980c-56d673e0d857-kube-api-access-655vv\") pod \"eeef5ae0-8dc9-452e-980c-56d673e0d857\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " Jul 13 14:55:54.846576 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.846518 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-podinfo\") pod \"eeef5ae0-8dc9-452e-980c-56d673e0d857\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " Jul 13 14:55:54.846576 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.846557 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeef5ae0-8dc9-452e-980c-56d673e0d857-istiod-ca-cert\") pod \"eeef5ae0-8dc9-452e-980c-56d673e0d857\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " Jul 13 14:55:54.846882 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.846616 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-credential-socket\") pod \"eeef5ae0-8dc9-452e-980c-56d673e0d857\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " Jul 13 14:55:54.846882 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.846808 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eeef5ae0-8dc9-452e-980c-56d673e0d857" (UID: "eeef5ae0-8dc9-452e-980c-56d673e0d857"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:55:54.846882 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.846835 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-workload-certs\") pod \"eeef5ae0-8dc9-452e-980c-56d673e0d857\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " Jul 13 14:55:54.847030 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.846887 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eeef5ae0-8dc9-452e-980c-56d673e0d857" (UID: "eeef5ae0-8dc9-452e-980c-56d673e0d857"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:55:54.847030 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.846903 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-envoy\") pod \"eeef5ae0-8dc9-452e-980c-56d673e0d857\" (UID: \"eeef5ae0-8dc9-452e-980c-56d673e0d857\") " Jul 13 14:55:54.847030 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.846988 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eeef5ae0-8dc9-452e-980c-56d673e0d857" (UID: "eeef5ae0-8dc9-452e-980c-56d673e0d857"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:55:54.847186 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.847048 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeef5ae0-8dc9-452e-980c-56d673e0d857-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eeef5ae0-8dc9-452e-980c-56d673e0d857" (UID: "eeef5ae0-8dc9-452e-980c-56d673e0d857"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:55:54.847317 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.847298 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-workload-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:55:54.847381 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.847323 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeef5ae0-8dc9-452e-980c-56d673e0d857-istiod-ca-cert\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:55:54.847381 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.847337 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-credential-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:55:54.847381 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.847351 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-workload-certs\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:55:54.847503 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.847401 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-data" (OuterVolumeSpecName: "istio-data") pod "eeef5ae0-8dc9-452e-980c-56d673e0d857" (UID: "eeef5ae0-8dc9-452e-980c-56d673e0d857"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:55:54.849167 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.849139 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-token" (OuterVolumeSpecName: "istio-token") pod "eeef5ae0-8dc9-452e-980c-56d673e0d857" (UID: "eeef5ae0-8dc9-452e-980c-56d673e0d857"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:55:54.849167 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.849145 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eeef5ae0-8dc9-452e-980c-56d673e0d857" (UID: "eeef5ae0-8dc9-452e-980c-56d673e0d857"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:55:54.849299 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.849204 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeef5ae0-8dc9-452e-980c-56d673e0d857-kube-api-access-655vv" (OuterVolumeSpecName: "kube-api-access-655vv") pod "eeef5ae0-8dc9-452e-980c-56d673e0d857" (UID: "eeef5ae0-8dc9-452e-980c-56d673e0d857"). InnerVolumeSpecName "kube-api-access-655vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:55:54.849299 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.849236 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eeef5ae0-8dc9-452e-980c-56d673e0d857" (UID: "eeef5ae0-8dc9-452e-980c-56d673e0d857"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 14:55:54.928260 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.928222 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:54.948214 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.948185 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-podinfo\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:55:54.948214 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.948211 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-envoy\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:55:54.948214 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.948221 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-data\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:55:54.948426 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.948229 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeef5ae0-8dc9-452e-980c-56d673e0d857-istio-token\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:55:54.948426 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:54.948237 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-655vv\" (UniqueName: \"kubernetes.io/projected/eeef5ae0-8dc9-452e-980c-56d673e0d857-kube-api-access-655vv\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:55:55.024724 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:55.024690 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:55.135151 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:55.135054 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:55:55.527932 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:55.527840 2566 generic.go:356] "Generic (PLEG): container finished" podID="eeef5ae0-8dc9-452e-980c-56d673e0d857" containerID="7ad847183de3eaf51522b084cb90c5bd1e2372e1d3df49fd43051eeef0071a79" exitCode=0 Jul 13 14:55:55.527932 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:55.527902 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" Jul 13 14:55:55.527932 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:55.527921 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" event={"ID":"eeef5ae0-8dc9-452e-980c-56d673e0d857","Type":"ContainerDied","Data":"7ad847183de3eaf51522b084cb90c5bd1e2372e1d3df49fd43051eeef0071a79"} Jul 13 14:55:55.528158 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:55.527996 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc" event={"ID":"eeef5ae0-8dc9-452e-980c-56d673e0d857","Type":"ContainerDied","Data":"6a56f2902b133dfdebc5db37f3b433e31272c9e923aa6fb4016be7eb2b2ced1d"} Jul 13 14:55:55.528158 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:55.528018 2566 scope.go:117] "RemoveContainer" containerID="7ad847183de3eaf51522b084cb90c5bd1e2372e1d3df49fd43051eeef0071a79" Jul 13 14:55:55.536689 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:55.536669 2566 scope.go:117] "RemoveContainer" containerID="de8774b83dd9ee38eb846dfbc1c2d8e9f55ad125621eccbd4a61e5085e7c2ee9" Jul 13 14:55:55.544526 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:55.544509 2566 scope.go:117] "RemoveContainer" containerID="7ad847183de3eaf51522b084cb90c5bd1e2372e1d3df49fd43051eeef0071a79" Jul 13 14:55:55.544775 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:55:55.544756 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ad847183de3eaf51522b084cb90c5bd1e2372e1d3df49fd43051eeef0071a79\": container with ID starting with 7ad847183de3eaf51522b084cb90c5bd1e2372e1d3df49fd43051eeef0071a79 not found: ID does not exist" containerID="7ad847183de3eaf51522b084cb90c5bd1e2372e1d3df49fd43051eeef0071a79" Jul 13 14:55:55.544852 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:55.544796 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ad847183de3eaf51522b084cb90c5bd1e2372e1d3df49fd43051eeef0071a79"} err="failed to get container status \"7ad847183de3eaf51522b084cb90c5bd1e2372e1d3df49fd43051eeef0071a79\": rpc error: code = NotFound desc = could not find container \"7ad847183de3eaf51522b084cb90c5bd1e2372e1d3df49fd43051eeef0071a79\": container with ID starting with 7ad847183de3eaf51522b084cb90c5bd1e2372e1d3df49fd43051eeef0071a79 not found: ID does not exist" Jul 13 14:55:55.544852 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:55.544817 2566 scope.go:117] "RemoveContainer" containerID="de8774b83dd9ee38eb846dfbc1c2d8e9f55ad125621eccbd4a61e5085e7c2ee9" Jul 13 14:55:55.545062 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:55:55.545038 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8774b83dd9ee38eb846dfbc1c2d8e9f55ad125621eccbd4a61e5085e7c2ee9\": container with ID starting with de8774b83dd9ee38eb846dfbc1c2d8e9f55ad125621eccbd4a61e5085e7c2ee9 not found: ID does not exist" containerID="de8774b83dd9ee38eb846dfbc1c2d8e9f55ad125621eccbd4a61e5085e7c2ee9" Jul 13 14:55:55.545193 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:55.545064 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8774b83dd9ee38eb846dfbc1c2d8e9f55ad125621eccbd4a61e5085e7c2ee9"} err="failed to get container status \"de8774b83dd9ee38eb846dfbc1c2d8e9f55ad125621eccbd4a61e5085e7c2ee9\": rpc error: code = NotFound desc = could not find container \"de8774b83dd9ee38eb846dfbc1c2d8e9f55ad125621eccbd4a61e5085e7c2ee9\": container with ID starting with de8774b83dd9ee38eb846dfbc1c2d8e9f55ad125621eccbd4a61e5085e7c2ee9 not found: ID does not exist" Jul 13 14:55:55.549345 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:55.549323 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc"] Jul 13 14:55:55.553330 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:55.553311 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3cced6c6-openshift-default-5bfdcf8687-7xqnc"] Jul 13 14:55:56.867145 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:55:56.867111 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeef5ae0-8dc9-452e-980c-56d673e0d857" path="/var/lib/kubelet/pods/eeef5ae0-8dc9-452e-980c-56d673e0d857/volumes" Jul 13 14:56:01.216207 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:01.216169 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:01.319302 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:01.319269 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:08.129834 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.129771 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:08.441720 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.441646 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6"] Jul 13 14:56:08.442067 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.442053 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeef5ae0-8dc9-452e-980c-56d673e0d857" containerName="istio-proxy" Jul 13 14:56:08.442130 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.442068 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeef5ae0-8dc9-452e-980c-56d673e0d857" containerName="istio-proxy" Jul 13 14:56:08.442130 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.442078 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4aa2b1d-0326-4177-8a37-1d5e8558b725" containerName="maas-api" Jul 13 14:56:08.442130 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.442083 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4aa2b1d-0326-4177-8a37-1d5e8558b725" containerName="maas-api" Jul 13 14:56:08.442130 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.442108 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeef5ae0-8dc9-452e-980c-56d673e0d857" containerName="istio-proxy" Jul 13 14:56:08.442130 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.442114 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeef5ae0-8dc9-452e-980c-56d673e0d857" containerName="istio-proxy" Jul 13 14:56:08.442286 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.442174 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4aa2b1d-0326-4177-8a37-1d5e8558b725" containerName="maas-api" Jul 13 14:56:08.442286 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.442186 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeef5ae0-8dc9-452e-980c-56d673e0d857" containerName="istio-proxy" Jul 13 14:56:08.451402 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.451373 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.454654 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.454632 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-575864a3-openshift-default-dockercfg-499m4\"" Jul 13 14:56:08.460602 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.460577 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6"] Jul 13 14:56:08.466798 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.466758 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:08.566394 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.566356 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-credential-socket\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.566580 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.566399 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-workload-socket\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.566580 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.566419 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c0e6292-af68-420a-a568-c4e96cf25184-istio-podinfo\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.566580 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.566434 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c0e6292-af68-420a-a568-c4e96cf25184-istio-token\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.566580 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.566524 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c0e6292-af68-420a-a568-c4e96cf25184-istiod-ca-cert\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.566824 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.566608 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4h5l\" (UniqueName: \"kubernetes.io/projected/8c0e6292-af68-420a-a568-c4e96cf25184-kube-api-access-h4h5l\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.566824 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.566675 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-workload-certs\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.566824 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.566700 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-istio-envoy\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.566824 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.566771 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-istio-data\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.668255 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.668219 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-workload-certs\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.668255 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.668252 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-istio-envoy\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.668480 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.668281 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-istio-data\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.668480 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.668328 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-credential-socket\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.668480 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.668365 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-workload-socket\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.668645 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.668520 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c0e6292-af68-420a-a568-c4e96cf25184-istio-podinfo\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.668645 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.668568 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c0e6292-af68-420a-a568-c4e96cf25184-istio-token\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.668645 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.668600 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c0e6292-af68-420a-a568-c4e96cf25184-istiod-ca-cert\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.668645 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.668619 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-workload-certs\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.668857 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.668690 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4h5l\" (UniqueName: \"kubernetes.io/projected/8c0e6292-af68-420a-a568-c4e96cf25184-kube-api-access-h4h5l\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.668857 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.668848 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-credential-socket\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.669132 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.669099 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-workload-socket\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.669281 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.669262 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-istio-data\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.669363 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.669345 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c0e6292-af68-420a-a568-c4e96cf25184-istiod-ca-cert\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.670687 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.670665 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-istio-envoy\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.671015 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.670997 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c0e6292-af68-420a-a568-c4e96cf25184-istio-podinfo\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.677579 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.677548 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c0e6292-af68-420a-a568-c4e96cf25184-istio-token\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.677777 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.677747 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4h5l\" (UniqueName: \"kubernetes.io/projected/8c0e6292-af68-420a-a568-c4e96cf25184-kube-api-access-h4h5l\") pod \"e2e-mt-575864a3-openshift-default-74b8695587-44lv6\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.762664 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.762581 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:08.860634 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.860584 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:08.903862 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.903838 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6"] Jul 13 14:56:08.905508 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:56:08.905479 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0e6292_af68_420a_a568_c4e96cf25184.slice/crio-5b71e7d80cc13662fd37c96e183e716904c2a7d6ce020cfeeb50b86d4a0c928a WatchSource:0}: Error finding container 5b71e7d80cc13662fd37c96e183e716904c2a7d6ce020cfeeb50b86d4a0c928a: Status 404 returned error can't find the container with id 5b71e7d80cc13662fd37c96e183e716904c2a7d6ce020cfeeb50b86d4a0c928a Jul 13 14:56:08.907591 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.907564 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:56:08.907669 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.907639 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:56:08.907707 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:08.907669 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:56:09.600556 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:09.600519 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" event={"ID":"8c0e6292-af68-420a-a568-c4e96cf25184","Type":"ContainerStarted","Data":"58b684b59049a4b4173b91d3aaf338c2281121c640a7ec5e5c5c5552ce788994"} Jul 13 14:56:09.600986 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:09.600578 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" event={"ID":"8c0e6292-af68-420a-a568-c4e96cf25184","Type":"ContainerStarted","Data":"5b71e7d80cc13662fd37c96e183e716904c2a7d6ce020cfeeb50b86d4a0c928a"} Jul 13 14:56:09.627643 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:09.627594 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" podStartSLOduration=1.62758018 podStartE2EDuration="1.62758018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:56:09.625178836 +0000 UTC m=+2111.309599153" watchObservedRunningTime="2026-07-13 14:56:09.62758018 +0000 UTC m=+2111.312000543" Jul 13 14:56:09.762868 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:09.762836 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:09.767609 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:09.767587 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:10.606110 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:10.606075 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:10.607100 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:10.607081 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:11.607621 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:11.607585 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:11.731396 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:11.731347 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:11.894639 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:11.894547 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q"] Jul 13 14:56:11.895128 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:11.895108 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeef5ae0-8dc9-452e-980c-56d673e0d857" containerName="istio-proxy" Jul 13 14:56:11.898258 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:11.898236 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:11.900767 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:11.900741 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-575864a3-serving-cert\"" Jul 13 14:56:11.900888 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:11.900749 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rxm62\"" Jul 13 14:56:11.909870 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:11.909844 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q"] Jul 13 14:56:11.925808 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:11.925757 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:12.002132 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:12.002100 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401ecdd3-267a-4ac2-ac6d-cd228bb59915-openshift-service-ca\") pod \"maas-api-e2e-mt-575864a3-5767745cf8-w259q\" (UID: \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:12.002312 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:12.002157 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401ecdd3-267a-4ac2-ac6d-cd228bb59915-maas-api-tls\") pod \"maas-api-e2e-mt-575864a3-5767745cf8-w259q\" (UID: \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:12.002312 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:12.002241 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79htn\" (UniqueName: \"kubernetes.io/projected/401ecdd3-267a-4ac2-ac6d-cd228bb59915-kube-api-access-79htn\") pod \"maas-api-e2e-mt-575864a3-5767745cf8-w259q\" (UID: \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:12.103220 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:12.103192 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401ecdd3-267a-4ac2-ac6d-cd228bb59915-openshift-service-ca\") pod \"maas-api-e2e-mt-575864a3-5767745cf8-w259q\" (UID: \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:12.103377 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:12.103247 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401ecdd3-267a-4ac2-ac6d-cd228bb59915-maas-api-tls\") pod \"maas-api-e2e-mt-575864a3-5767745cf8-w259q\" (UID: \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:12.103377 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:56:12.103331 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-serving-cert: secret "maas-api-e2e-mt-575864a3-serving-cert" not found Jul 13 14:56:12.103377 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:12.103360 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79htn\" (UniqueName: \"kubernetes.io/projected/401ecdd3-267a-4ac2-ac6d-cd228bb59915-kube-api-access-79htn\") pod \"maas-api-e2e-mt-575864a3-5767745cf8-w259q\" (UID: \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:12.103502 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:56:12.103386 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/401ecdd3-267a-4ac2-ac6d-cd228bb59915-maas-api-tls podName:401ecdd3-267a-4ac2-ac6d-cd228bb59915 nodeName:}" failed. No retries permitted until 2026-07-13 14:56:12.603369691 +0000 UTC m=+2114.287789975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/401ecdd3-267a-4ac2-ac6d-cd228bb59915-maas-api-tls") pod "maas-api-e2e-mt-575864a3-5767745cf8-w259q" (UID: "401ecdd3-267a-4ac2-ac6d-cd228bb59915") : secret "maas-api-e2e-mt-575864a3-serving-cert" not found Jul 13 14:56:12.103923 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:12.103904 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401ecdd3-267a-4ac2-ac6d-cd228bb59915-openshift-service-ca\") pod \"maas-api-e2e-mt-575864a3-5767745cf8-w259q\" (UID: \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:12.114325 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:12.114301 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79htn\" (UniqueName: \"kubernetes.io/projected/401ecdd3-267a-4ac2-ac6d-cd228bb59915-kube-api-access-79htn\") pod \"maas-api-e2e-mt-575864a3-5767745cf8-w259q\" (UID: \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:12.417719 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:12.417687 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:12.522214 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:12.522182 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:12.608584 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:12.608555 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401ecdd3-267a-4ac2-ac6d-cd228bb59915-maas-api-tls\") pod \"maas-api-e2e-mt-575864a3-5767745cf8-w259q\" (UID: \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:12.610888 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:12.610871 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401ecdd3-267a-4ac2-ac6d-cd228bb59915-maas-api-tls\") pod \"maas-api-e2e-mt-575864a3-5767745cf8-w259q\" (UID: \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:12.617986 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:12.617948 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:12.810315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:12.810229 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:12.943835 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:12.943803 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q"] Jul 13 14:56:12.944879 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:56:12.944852 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401ecdd3_267a_4ac2_ac6d_cd228bb59915.slice/crio-cfb06609ce880364dd0bf81f3edb9f78a228f37292c1b4c62034f037407392d8 WatchSource:0}: Error finding container cfb06609ce880364dd0bf81f3edb9f78a228f37292c1b4c62034f037407392d8: Status 404 returned error can't find the container with id cfb06609ce880364dd0bf81f3edb9f78a228f37292c1b4c62034f037407392d8 Jul 13 14:56:13.625289 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:13.625257 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" event={"ID":"401ecdd3-267a-4ac2-ac6d-cd228bb59915","Type":"ContainerStarted","Data":"cfb06609ce880364dd0bf81f3edb9f78a228f37292c1b4c62034f037407392d8"} Jul 13 14:56:14.632308 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:14.632265 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" event={"ID":"401ecdd3-267a-4ac2-ac6d-cd228bb59915","Type":"ContainerStarted","Data":"00746b59052a8f7ae8a758a028089890701457db18bfb6aeb90ed1b9efe46115"} Jul 13 14:56:14.632775 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:14.632333 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:14.652513 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:14.652469 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" podStartSLOduration=2.61118168 podStartE2EDuration="3.65245205s" totalImagesPullingTime="1.04127037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:56:14.651070567 +0000 UTC m=+2116.335490896" watchObservedRunningTime="2026-07-13 14:56:14.65245205 +0000 UTC m=+2116.336872355" Jul 13 14:56:16.154391 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:16.154354 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:20.649111 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:20.649078 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:23.334016 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:23.333983 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:26.102483 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:26.101850 2566 patch_prober.go:28] interesting pod/e2e-mt-575864a3-openshift-default-74b8695587-44lv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": read tcp 10.133.0.2:43968->10.133.0.64:15021: read: connection reset by peer" start-of-body= Jul 13 14:56:26.102483 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:26.101918 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" podUID="8c0e6292-af68-420a-a568-c4e96cf25184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": read tcp 10.133.0.2:43968->10.133.0.64:15021: read: connection reset by peer" Jul 13 14:56:26.701721 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:26.701687 2566 generic.go:356] "Generic (PLEG): container finished" podID="8c0e6292-af68-420a-a568-c4e96cf25184" containerID="58b684b59049a4b4173b91d3aaf338c2281121c640a7ec5e5c5c5552ce788994" exitCode=0 Jul 13 14:56:26.701917 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:26.701752 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" event={"ID":"8c0e6292-af68-420a-a568-c4e96cf25184","Type":"ContainerDied","Data":"58b684b59049a4b4173b91d3aaf338c2281121c640a7ec5e5c5c5552ce788994"} Jul 13 14:56:26.702181 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:26.702166 2566 scope.go:117] "RemoveContainer" containerID="58b684b59049a4b4173b91d3aaf338c2281121c640a7ec5e5c5c5552ce788994" Jul 13 14:56:26.703177 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:26.703151 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:56:26.703248 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:26.703217 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:56:26.703286 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:26.703249 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:56:27.709495 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:27.709462 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" event={"ID":"8c0e6292-af68-420a-a568-c4e96cf25184","Type":"ContainerStarted","Data":"8a59fa647627fade235636f6ebd475b38b1ae0df9c23059fa3fbe9ae874069c5"} Jul 13 14:56:27.763149 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:27.763107 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:27.763392 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:27.763368 2566 patch_prober.go:28] interesting pod/e2e-mt-575864a3-openshift-default-74b8695587-44lv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 13 14:56:27.763483 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:27.763426 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" podUID="8c0e6292-af68-420a-a568-c4e96cf25184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 13 14:56:28.763183 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:28.763147 2566 patch_prober.go:28] interesting pod/e2e-mt-575864a3-openshift-default-74b8695587-44lv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 13 14:56:28.763541 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:28.763196 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" podUID="8c0e6292-af68-420a-a568-c4e96cf25184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 13 14:56:29.763309 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:29.763271 2566 patch_prober.go:28] interesting pod/e2e-mt-575864a3-openshift-default-74b8695587-44lv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 13 14:56:29.763804 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:29.763347 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" podUID="8c0e6292-af68-420a-a568-c4e96cf25184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 13 14:56:29.940691 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:29.940646 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:30.043489 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:30.043340 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:30.763237 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:30.763193 2566 patch_prober.go:28] interesting pod/e2e-mt-575864a3-openshift-default-74b8695587-44lv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 13 14:56:30.763453 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:30.763272 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" podUID="8c0e6292-af68-420a-a568-c4e96cf25184" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 13 14:56:31.766517 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:31.766485 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:31.767035 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:31.766881 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:31.767655 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:31.767625 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:56:48.522900 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.522864 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q"] Jul 13 14:56:48.523438 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.523132 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" podUID="401ecdd3-267a-4ac2-ac6d-cd228bb59915" containerName="maas-api" containerID="cri-o://00746b59052a8f7ae8a758a028089890701457db18bfb6aeb90ed1b9efe46115" gracePeriod=30 Jul 13 14:56:48.546420 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.546388 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:48.606595 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.606567 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:48.777826 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.777748 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:48.830443 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.830409 2566 generic.go:356] "Generic (PLEG): container finished" podID="401ecdd3-267a-4ac2-ac6d-cd228bb59915" containerID="00746b59052a8f7ae8a758a028089890701457db18bfb6aeb90ed1b9efe46115" exitCode=0 Jul 13 14:56:48.830639 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.830453 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" event={"ID":"401ecdd3-267a-4ac2-ac6d-cd228bb59915","Type":"ContainerDied","Data":"00746b59052a8f7ae8a758a028089890701457db18bfb6aeb90ed1b9efe46115"} Jul 13 14:56:48.830639 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.830467 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" Jul 13 14:56:48.830639 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.830527 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q" event={"ID":"401ecdd3-267a-4ac2-ac6d-cd228bb59915","Type":"ContainerDied","Data":"cfb06609ce880364dd0bf81f3edb9f78a228f37292c1b4c62034f037407392d8"} Jul 13 14:56:48.830639 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.830565 2566 scope.go:117] "RemoveContainer" containerID="00746b59052a8f7ae8a758a028089890701457db18bfb6aeb90ed1b9efe46115" Jul 13 14:56:48.840341 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.840325 2566 scope.go:117] "RemoveContainer" containerID="00746b59052a8f7ae8a758a028089890701457db18bfb6aeb90ed1b9efe46115" Jul 13 14:56:48.840598 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:56:48.840580 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00746b59052a8f7ae8a758a028089890701457db18bfb6aeb90ed1b9efe46115\": container with ID starting with 00746b59052a8f7ae8a758a028089890701457db18bfb6aeb90ed1b9efe46115 not found: ID does not exist" containerID="00746b59052a8f7ae8a758a028089890701457db18bfb6aeb90ed1b9efe46115" Jul 13 14:56:48.840669 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.840611 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00746b59052a8f7ae8a758a028089890701457db18bfb6aeb90ed1b9efe46115"} err="failed to get container status \"00746b59052a8f7ae8a758a028089890701457db18bfb6aeb90ed1b9efe46115\": rpc error: code = NotFound desc = could not find container \"00746b59052a8f7ae8a758a028089890701457db18bfb6aeb90ed1b9efe46115\": container with ID starting with 00746b59052a8f7ae8a758a028089890701457db18bfb6aeb90ed1b9efe46115 not found: ID does not exist" Jul 13 14:56:48.930865 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.930828 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401ecdd3-267a-4ac2-ac6d-cd228bb59915-openshift-service-ca\") pod \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\" (UID: \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\") " Jul 13 14:56:48.931047 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.931030 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79htn\" (UniqueName: \"kubernetes.io/projected/401ecdd3-267a-4ac2-ac6d-cd228bb59915-kube-api-access-79htn\") pod \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\" (UID: \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\") " Jul 13 14:56:48.931130 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.931104 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401ecdd3-267a-4ac2-ac6d-cd228bb59915-maas-api-tls\") pod \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\" (UID: \"401ecdd3-267a-4ac2-ac6d-cd228bb59915\") " Jul 13 14:56:48.931297 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.931270 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401ecdd3-267a-4ac2-ac6d-cd228bb59915-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "401ecdd3-267a-4ac2-ac6d-cd228bb59915" (UID: "401ecdd3-267a-4ac2-ac6d-cd228bb59915"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:56:48.931430 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.931412 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401ecdd3-267a-4ac2-ac6d-cd228bb59915-openshift-service-ca\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:56:48.933163 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.933132 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401ecdd3-267a-4ac2-ac6d-cd228bb59915-kube-api-access-79htn" (OuterVolumeSpecName: "kube-api-access-79htn") pod "401ecdd3-267a-4ac2-ac6d-cd228bb59915" (UID: "401ecdd3-267a-4ac2-ac6d-cd228bb59915"). InnerVolumeSpecName "kube-api-access-79htn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:56:48.933604 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:48.933582 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401ecdd3-267a-4ac2-ac6d-cd228bb59915-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "401ecdd3-267a-4ac2-ac6d-cd228bb59915" (UID: "401ecdd3-267a-4ac2-ac6d-cd228bb59915"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 14:56:49.032652 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:49.032561 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401ecdd3-267a-4ac2-ac6d-cd228bb59915-maas-api-tls\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:56:49.032652 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:49.032600 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79htn\" (UniqueName: \"kubernetes.io/projected/401ecdd3-267a-4ac2-ac6d-cd228bb59915-kube-api-access-79htn\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:56:49.155636 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:49.155607 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q"] Jul 13 14:56:49.157895 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:49.157873 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-575864a3-5767745cf8-w259q"] Jul 13 14:56:49.559890 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:49.559848 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:56:50.867684 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:50.867605 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="401ecdd3-267a-4ac2-ac6d-cd228bb59915" path="/var/lib/kubelet/pods/401ecdd3-267a-4ac2-ac6d-cd228bb59915/volumes" Jul 13 14:56:54.583928 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:56:54.583892 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:38.299179 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:38.299143 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:38.531821 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:38.531763 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:45.046872 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:45.046835 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:45.128632 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:45.128578 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:45.325328 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:45.325298 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6"] Jul 13 14:57:45.325523 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:45.325502 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" podUID="8c0e6292-af68-420a-a568-c4e96cf25184" containerName="istio-proxy" containerID="cri-o://8a59fa647627fade235636f6ebd475b38b1ae0df9c23059fa3fbe9ae874069c5" gracePeriod=30 Jul 13 14:57:45.342549 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:45.342518 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:45.625696 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:45.625611 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:46.719961 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:46.719918 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:46.769398 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:46.769361 2566 patch_prober.go:28] interesting pod/e2e-mt-575864a3-openshift-default-74b8695587-44lv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 14:57:46.769578 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:46.769434 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" podUID="8c0e6292-af68-420a-a568-c4e96cf25184" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 14:57:46.819197 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:46.819151 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:46.917614 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:46.917578 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:47.698185 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:47.698142 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:47.828021 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:47.827985 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:47.930745 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:47.930704 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:49.351735 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:49.351699 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:50.572569 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.572543 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:57:50.594058 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.593979 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-istio-envoy\") pod \"8c0e6292-af68-420a-a568-c4e96cf25184\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " Jul 13 14:57:50.594058 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.594035 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-credential-socket\") pod \"8c0e6292-af68-420a-a568-c4e96cf25184\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " Jul 13 14:57:50.594058 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.594058 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-workload-socket\") pod \"8c0e6292-af68-420a-a568-c4e96cf25184\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " Jul 13 14:57:50.594290 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.594094 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-istio-data\") pod \"8c0e6292-af68-420a-a568-c4e96cf25184\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " Jul 13 14:57:50.594290 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.594119 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-workload-certs\") pod \"8c0e6292-af68-420a-a568-c4e96cf25184\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " Jul 13 14:57:50.594290 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.594157 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c0e6292-af68-420a-a568-c4e96cf25184-istiod-ca-cert\") pod \"8c0e6292-af68-420a-a568-c4e96cf25184\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " Jul 13 14:57:50.594290 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.594195 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4h5l\" (UniqueName: \"kubernetes.io/projected/8c0e6292-af68-420a-a568-c4e96cf25184-kube-api-access-h4h5l\") pod \"8c0e6292-af68-420a-a568-c4e96cf25184\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " Jul 13 14:57:50.594501 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.594380 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8c0e6292-af68-420a-a568-c4e96cf25184" (UID: "8c0e6292-af68-420a-a568-c4e96cf25184"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:57:50.594501 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.594395 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8c0e6292-af68-420a-a568-c4e96cf25184" (UID: "8c0e6292-af68-420a-a568-c4e96cf25184"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:57:50.594501 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.594425 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8c0e6292-af68-420a-a568-c4e96cf25184" (UID: "8c0e6292-af68-420a-a568-c4e96cf25184"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:57:50.594647 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.594553 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-credential-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:57:50.594647 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.594574 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-workload-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:57:50.594647 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.594591 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-workload-certs\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:57:50.594647 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.594620 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0e6292-af68-420a-a568-c4e96cf25184-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8c0e6292-af68-420a-a568-c4e96cf25184" (UID: "8c0e6292-af68-420a-a568-c4e96cf25184"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 14:57:50.594923 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.594904 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-istio-data" (OuterVolumeSpecName: "istio-data") pod "8c0e6292-af68-420a-a568-c4e96cf25184" (UID: "8c0e6292-af68-420a-a568-c4e96cf25184"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:57:50.596053 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.596026 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8c0e6292-af68-420a-a568-c4e96cf25184" (UID: "8c0e6292-af68-420a-a568-c4e96cf25184"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 14:57:50.596308 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.596291 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0e6292-af68-420a-a568-c4e96cf25184-kube-api-access-h4h5l" (OuterVolumeSpecName: "kube-api-access-h4h5l") pod "8c0e6292-af68-420a-a568-c4e96cf25184" (UID: "8c0e6292-af68-420a-a568-c4e96cf25184"). InnerVolumeSpecName "kube-api-access-h4h5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:57:50.695369 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.695339 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c0e6292-af68-420a-a568-c4e96cf25184-istio-token\") pod \"8c0e6292-af68-420a-a568-c4e96cf25184\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " Jul 13 14:57:50.695369 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.695375 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c0e6292-af68-420a-a568-c4e96cf25184-istio-podinfo\") pod \"8c0e6292-af68-420a-a568-c4e96cf25184\" (UID: \"8c0e6292-af68-420a-a568-c4e96cf25184\") " Jul 13 14:57:50.695606 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.695589 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-istio-data\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:57:50.695668 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.695606 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c0e6292-af68-420a-a568-c4e96cf25184-istiod-ca-cert\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:57:50.695668 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.695620 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4h5l\" (UniqueName: \"kubernetes.io/projected/8c0e6292-af68-420a-a568-c4e96cf25184-kube-api-access-h4h5l\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:57:50.695771 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.695674 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c0e6292-af68-420a-a568-c4e96cf25184-istio-envoy\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:57:50.697468 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.697445 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0e6292-af68-420a-a568-c4e96cf25184-istio-token" (OuterVolumeSpecName: "istio-token") pod "8c0e6292-af68-420a-a568-c4e96cf25184" (UID: "8c0e6292-af68-420a-a568-c4e96cf25184"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 14:57:50.697552 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.697473 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c0e6292-af68-420a-a568-c4e96cf25184-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8c0e6292-af68-420a-a568-c4e96cf25184" (UID: "8c0e6292-af68-420a-a568-c4e96cf25184"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 14:57:50.796279 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.796243 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c0e6292-af68-420a-a568-c4e96cf25184-istio-token\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:57:50.796279 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:50.796272 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c0e6292-af68-420a-a568-c4e96cf25184-istio-podinfo\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 14:57:51.160151 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:51.160062 2566 generic.go:356] "Generic (PLEG): container finished" podID="8c0e6292-af68-420a-a568-c4e96cf25184" containerID="8a59fa647627fade235636f6ebd475b38b1ae0df9c23059fa3fbe9ae874069c5" exitCode=0 Jul 13 14:57:51.160151 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:51.160123 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" Jul 13 14:57:51.160334 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:51.160162 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" event={"ID":"8c0e6292-af68-420a-a568-c4e96cf25184","Type":"ContainerDied","Data":"8a59fa647627fade235636f6ebd475b38b1ae0df9c23059fa3fbe9ae874069c5"} Jul 13 14:57:51.160334 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:51.160225 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6" event={"ID":"8c0e6292-af68-420a-a568-c4e96cf25184","Type":"ContainerDied","Data":"5b71e7d80cc13662fd37c96e183e716904c2a7d6ce020cfeeb50b86d4a0c928a"} Jul 13 14:57:51.160334 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:51.160251 2566 scope.go:117] "RemoveContainer" containerID="8a59fa647627fade235636f6ebd475b38b1ae0df9c23059fa3fbe9ae874069c5" Jul 13 14:57:51.169132 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:51.169114 2566 scope.go:117] "RemoveContainer" containerID="58b684b59049a4b4173b91d3aaf338c2281121c640a7ec5e5c5c5552ce788994" Jul 13 14:57:51.177017 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:51.177001 2566 scope.go:117] "RemoveContainer" containerID="8a59fa647627fade235636f6ebd475b38b1ae0df9c23059fa3fbe9ae874069c5" Jul 13 14:57:51.177289 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:57:51.177267 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a59fa647627fade235636f6ebd475b38b1ae0df9c23059fa3fbe9ae874069c5\": container with ID starting with 8a59fa647627fade235636f6ebd475b38b1ae0df9c23059fa3fbe9ae874069c5 not found: ID does not exist" containerID="8a59fa647627fade235636f6ebd475b38b1ae0df9c23059fa3fbe9ae874069c5" Jul 13 14:57:51.177370 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:51.177295 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a59fa647627fade235636f6ebd475b38b1ae0df9c23059fa3fbe9ae874069c5"} err="failed to get container status \"8a59fa647627fade235636f6ebd475b38b1ae0df9c23059fa3fbe9ae874069c5\": rpc error: code = NotFound desc = could not find container \"8a59fa647627fade235636f6ebd475b38b1ae0df9c23059fa3fbe9ae874069c5\": container with ID starting with 8a59fa647627fade235636f6ebd475b38b1ae0df9c23059fa3fbe9ae874069c5 not found: ID does not exist" Jul 13 14:57:51.177370 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:51.177312 2566 scope.go:117] "RemoveContainer" containerID="58b684b59049a4b4173b91d3aaf338c2281121c640a7ec5e5c5c5552ce788994" Jul 13 14:57:51.177556 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:57:51.177541 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b684b59049a4b4173b91d3aaf338c2281121c640a7ec5e5c5c5552ce788994\": container with ID starting with 58b684b59049a4b4173b91d3aaf338c2281121c640a7ec5e5c5c5552ce788994 not found: ID does not exist" containerID="58b684b59049a4b4173b91d3aaf338c2281121c640a7ec5e5c5c5552ce788994" Jul 13 14:57:51.177597 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:51.177562 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b684b59049a4b4173b91d3aaf338c2281121c640a7ec5e5c5c5552ce788994"} err="failed to get container status \"58b684b59049a4b4173b91d3aaf338c2281121c640a7ec5e5c5c5552ce788994\": rpc error: code = NotFound desc = could not find container \"58b684b59049a4b4173b91d3aaf338c2281121c640a7ec5e5c5c5552ce788994\": container with ID starting with 58b684b59049a4b4173b91d3aaf338c2281121c640a7ec5e5c5c5552ce788994 not found: ID does not exist" Jul 13 14:57:51.182467 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:51.182439 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6"] Jul 13 14:57:51.186301 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:51.186282 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-575864a3-openshift-default-74b8695587-44lv6"] Jul 13 14:57:52.869837 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:52.869801 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c0e6292-af68-420a-a568-c4e96cf25184" path="/var/lib/kubelet/pods/8c0e6292-af68-420a-a568-c4e96cf25184/volumes" Jul 13 14:57:55.528981 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:55.528936 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:55.592369 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:55.592335 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:57:56.237667 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:57:56.237626 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:02.807845 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:02.807810 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:02.871537 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:02.871501 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:02.956290 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:02.956256 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:03.050123 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:03.050090 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:13.558211 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:13.558170 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:13.950436 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:13.950398 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:23.496060 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:23.495972 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:23.560590 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:23.560547 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:34.916937 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:34.916902 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:41.146642 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.146598 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:41.219743 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.219704 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:41.498887 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.498774 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:41.829306 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.829272 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6"] Jul 13 14:58:41.829890 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.829870 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c0e6292-af68-420a-a568-c4e96cf25184" containerName="istio-proxy" Jul 13 14:58:41.829994 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.829892 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c0e6292-af68-420a-a568-c4e96cf25184" containerName="istio-proxy" Jul 13 14:58:41.829994 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.829907 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c0e6292-af68-420a-a568-c4e96cf25184" containerName="istio-proxy" Jul 13 14:58:41.829994 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.829915 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c0e6292-af68-420a-a568-c4e96cf25184" containerName="istio-proxy" Jul 13 14:58:41.829994 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.829940 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401ecdd3-267a-4ac2-ac6d-cd228bb59915" containerName="maas-api" Jul 13 14:58:41.829994 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.829948 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="401ecdd3-267a-4ac2-ac6d-cd228bb59915" containerName="maas-api" Jul 13 14:58:41.830187 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.830023 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c0e6292-af68-420a-a568-c4e96cf25184" containerName="istio-proxy" Jul 13 14:58:41.830187 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.830037 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c0e6292-af68-420a-a568-c4e96cf25184" containerName="istio-proxy" Jul 13 14:58:41.830187 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.830050 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="401ecdd3-267a-4ac2-ac6d-cd228bb59915" containerName="maas-api" Jul 13 14:58:41.833394 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.833374 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:41.837228 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.837207 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d6591f-openshift-default-dockercfg-pgkq4\"" Jul 13 14:58:41.848439 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.848397 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:41.848972 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.848936 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6"] Jul 13 14:58:41.946781 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.946750 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-workload-certs\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:41.946953 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.946808 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-istio-data\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:41.946953 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.946851 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-workload-socket\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:41.946953 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.946869 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ea89bf-0800-4402-96f1-945b197af8de-istio-token\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:41.946953 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.946892 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-credential-socket\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:41.946953 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.946912 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxc9r\" (UniqueName: \"kubernetes.io/projected/58ea89bf-0800-4402-96f1-945b197af8de-kube-api-access-pxc9r\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:41.946953 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.946943 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ea89bf-0800-4402-96f1-945b197af8de-istio-podinfo\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:41.947168 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.946989 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-istio-envoy\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:41.947168 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.947018 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ea89bf-0800-4402-96f1-945b197af8de-istiod-ca-cert\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:41.985796 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:41.985765 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:42.048333 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.048299 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ea89bf-0800-4402-96f1-945b197af8de-istio-podinfo\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.048333 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.048333 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-istio-envoy\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.048555 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.048352 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ea89bf-0800-4402-96f1-945b197af8de-istiod-ca-cert\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.048618 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.048550 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-workload-certs\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.048618 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.048590 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-istio-data\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.048726 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.048649 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-workload-socket\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.048726 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.048680 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ea89bf-0800-4402-96f1-945b197af8de-istio-token\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.048726 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.048711 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-credential-socket\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.048903 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.048740 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxc9r\" (UniqueName: \"kubernetes.io/projected/58ea89bf-0800-4402-96f1-945b197af8de-kube-api-access-pxc9r\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.048964 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.048902 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-workload-certs\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.049016 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.048988 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-istio-data\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.049224 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.049198 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ea89bf-0800-4402-96f1-945b197af8de-istiod-ca-cert\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.049325 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.049255 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-credential-socket\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.049325 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.049299 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-workload-socket\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.050716 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.050691 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-istio-envoy\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.051148 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.051127 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ea89bf-0800-4402-96f1-945b197af8de-istio-podinfo\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.058651 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.058627 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ea89bf-0800-4402-96f1-945b197af8de-istio-token\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.059514 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.059493 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxc9r\" (UniqueName: \"kubernetes.io/projected/58ea89bf-0800-4402-96f1-945b197af8de-kube-api-access-pxc9r\") pod \"e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.145030 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.144953 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:42.283866 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.283834 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6"] Jul 13 14:58:42.286897 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:58:42.286871 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ea89bf_0800_4402_96f1_945b197af8de.slice/crio-d2b0d58ee2b30a74969c718040bd903056781389ae867e44a98994fa09bf88a3 WatchSource:0}: Error finding container d2b0d58ee2b30a74969c718040bd903056781389ae867e44a98994fa09bf88a3: Status 404 returned error can't find the container with id d2b0d58ee2b30a74969c718040bd903056781389ae867e44a98994fa09bf88a3 Jul 13 14:58:42.289035 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.289002 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:58:42.289123 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.289070 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:58:42.289123 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.289096 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:58:42.440086 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.439987 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" event={"ID":"58ea89bf-0800-4402-96f1-945b197af8de","Type":"ContainerStarted","Data":"4a31a17c16347036914bb4eb90fb96fccad26137e46fce0eeb62c5c686881e20"} Jul 13 14:58:42.440086 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.440058 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" event={"ID":"58ea89bf-0800-4402-96f1-945b197af8de","Type":"ContainerStarted","Data":"d2b0d58ee2b30a74969c718040bd903056781389ae867e44a98994fa09bf88a3"} Jul 13 14:58:42.463398 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:42.463350 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" podStartSLOduration=1.463332475 podStartE2EDuration="1.463332475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:58:42.461599703 +0000 UTC m=+2264.146020009" watchObservedRunningTime="2026-07-13 14:58:42.463332475 +0000 UTC m=+2264.147752781" Jul 13 14:58:43.145480 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:43.145451 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:43.150435 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:43.150411 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:43.445132 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:43.445047 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:43.445968 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:43.445950 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:58:44.465311 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.465274 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:44.559647 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.559604 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:44.667007 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.666975 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz"] Jul 13 14:58:44.670559 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.670539 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 14:58:44.673153 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.673125 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rxm62\"" Jul 13 14:58:44.673293 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.673171 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d6591f-serving-cert\"" Jul 13 14:58:44.684927 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.684903 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz"] Jul 13 14:58:44.728198 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.728122 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:44.775485 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.775458 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6rzz\" (UniqueName: \"kubernetes.io/projected/8f01234c-ec08-4a24-8fe7-41ab4198268b-kube-api-access-w6rzz\") pod \"maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz\" (UID: \"8f01234c-ec08-4a24-8fe7-41ab4198268b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 14:58:44.775653 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.775498 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f01234c-ec08-4a24-8fe7-41ab4198268b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz\" (UID: \"8f01234c-ec08-4a24-8fe7-41ab4198268b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 14:58:44.775653 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.775520 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f01234c-ec08-4a24-8fe7-41ab4198268b-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz\" (UID: \"8f01234c-ec08-4a24-8fe7-41ab4198268b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 14:58:44.876688 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.876661 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6rzz\" (UniqueName: \"kubernetes.io/projected/8f01234c-ec08-4a24-8fe7-41ab4198268b-kube-api-access-w6rzz\") pod \"maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz\" (UID: \"8f01234c-ec08-4a24-8fe7-41ab4198268b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 14:58:44.876860 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.876701 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f01234c-ec08-4a24-8fe7-41ab4198268b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz\" (UID: \"8f01234c-ec08-4a24-8fe7-41ab4198268b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 14:58:44.876860 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.876719 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f01234c-ec08-4a24-8fe7-41ab4198268b-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz\" (UID: \"8f01234c-ec08-4a24-8fe7-41ab4198268b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 14:58:44.876940 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:58:44.876861 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-serving-cert: secret "maas-api-e2e-inf-a-d6591f-serving-cert" not found Jul 13 14:58:44.876940 ip-10-0-131-188 kubenswrapper[2566]: E0713 14:58:44.876919 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f01234c-ec08-4a24-8fe7-41ab4198268b-maas-api-tls podName:8f01234c-ec08-4a24-8fe7-41ab4198268b nodeName:}" failed. No retries permitted until 2026-07-13 14:58:45.376899425 +0000 UTC m=+2267.061319712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f01234c-ec08-4a24-8fe7-41ab4198268b-maas-api-tls") pod "maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" (UID: "8f01234c-ec08-4a24-8fe7-41ab4198268b") : secret "maas-api-e2e-inf-a-d6591f-serving-cert" not found Jul 13 14:58:44.877382 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.877363 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f01234c-ec08-4a24-8fe7-41ab4198268b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz\" (UID: \"8f01234c-ec08-4a24-8fe7-41ab4198268b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 14:58:44.887390 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.887360 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6rzz\" (UniqueName: \"kubernetes.io/projected/8f01234c-ec08-4a24-8fe7-41ab4198268b-kube-api-access-w6rzz\") pod \"maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz\" (UID: \"8f01234c-ec08-4a24-8fe7-41ab4198268b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 14:58:44.996713 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:44.996631 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:45.259293 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:45.259196 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:45.356479 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:45.356440 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:45.381552 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:45.381524 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f01234c-ec08-4a24-8fe7-41ab4198268b-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz\" (UID: \"8f01234c-ec08-4a24-8fe7-41ab4198268b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 14:58:45.384001 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:45.383985 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f01234c-ec08-4a24-8fe7-41ab4198268b-maas-api-tls\") pod \"maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz\" (UID: \"8f01234c-ec08-4a24-8fe7-41ab4198268b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 14:58:45.583712 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:45.583676 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 14:58:45.721488 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:45.721458 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz"] Jul 13 14:58:45.724197 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:58:45.724165 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f01234c_ec08_4a24_8fe7_41ab4198268b.slice/crio-f93e675b72fc0d7acd40f1c4fa3140e340aee36f9b31400093ec37034b9dd596 WatchSource:0}: Error finding container f93e675b72fc0d7acd40f1c4fa3140e340aee36f9b31400093ec37034b9dd596: Status 404 returned error can't find the container with id f93e675b72fc0d7acd40f1c4fa3140e340aee36f9b31400093ec37034b9dd596 Jul 13 14:58:45.725732 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:45.725713 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 14:58:46.463760 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:46.463659 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" event={"ID":"8f01234c-ec08-4a24-8fe7-41ab4198268b","Type":"ContainerStarted","Data":"23531a5aa6ee814563376a3bb7a318cfd759e3692cd9f841caea7c1f84d2213d"} Jul 13 14:58:46.463760 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:46.463730 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" event={"ID":"8f01234c-ec08-4a24-8fe7-41ab4198268b","Type":"ContainerStarted","Data":"f93e675b72fc0d7acd40f1c4fa3140e340aee36f9b31400093ec37034b9dd596"} Jul 13 14:58:46.463760 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:46.463761 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 14:58:46.484869 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:46.484819 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" podStartSLOduration=2.04827521 podStartE2EDuration="2.484802145s" totalImagesPullingTime="436.526935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:58:46.482243811 +0000 UTC m=+2268.166664117" watchObservedRunningTime="2026-07-13 14:58:46.484802145 +0000 UTC m=+2268.169222449" Jul 13 14:58:48.973610 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:48.973580 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:49.608650 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.608614 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb"] Jul 13 14:58:49.613011 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.612991 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.615901 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.615879 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b132ed-openshift-default-dockercfg-6cdbc\"" Jul 13 14:58:49.625572 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.625551 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb"] Jul 13 14:58:49.720445 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.720411 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-workload-certs\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.720445 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.720449 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swmb2\" (UniqueName: \"kubernetes.io/projected/aacd798b-5303-4259-9536-c4e3e1f542e6-kube-api-access-swmb2\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.720656 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.720582 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-token\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.720656 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.720620 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-podinfo\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.720656 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.720646 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-credential-socket\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.720768 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.720741 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-workload-socket\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.720768 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.720764 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-envoy\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.720881 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.720779 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-data\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.720881 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.720861 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacd798b-5303-4259-9536-c4e3e1f542e6-istiod-ca-cert\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.822315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.822281 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-workload-socket\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.822315 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.822318 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-envoy\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.822580 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.822335 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-data\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.822580 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.822471 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacd798b-5303-4259-9536-c4e3e1f542e6-istiod-ca-cert\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.822580 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.822525 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-workload-certs\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.822580 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.822552 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swmb2\" (UniqueName: \"kubernetes.io/projected/aacd798b-5303-4259-9536-c4e3e1f542e6-kube-api-access-swmb2\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.822799 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.822662 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-token\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.822799 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.822695 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-podinfo\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.822799 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.822699 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-data\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.822799 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.822732 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-credential-socket\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.822799 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.822756 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-workload-socket\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.823081 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.822894 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-workload-certs\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.823147 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.823122 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-credential-socket\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.823191 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.823170 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacd798b-5303-4259-9536-c4e3e1f542e6-istiod-ca-cert\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.824651 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.824629 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-envoy\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.825054 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.825039 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-podinfo\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.832592 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.832568 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-token\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.833202 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.833178 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swmb2\" (UniqueName: \"kubernetes.io/projected/aacd798b-5303-4259-9536-c4e3e1f542e6-kube-api-access-swmb2\") pod \"e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:49.924165 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:49.924082 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:50.075934 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:58:50.075903 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacd798b_5303_4259_9536_c4e3e1f542e6.slice/crio-73aa6f6ef2bccca58889ce01b5209806639b0fff27a0811cb1d7873d901a58b3 WatchSource:0}: Error finding container 73aa6f6ef2bccca58889ce01b5209806639b0fff27a0811cb1d7873d901a58b3: Status 404 returned error can't find the container with id 73aa6f6ef2bccca58889ce01b5209806639b0fff27a0811cb1d7873d901a58b3 Jul 13 14:58:50.076909 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:50.076878 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb"] Jul 13 14:58:50.078357 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:50.078327 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:58:50.078450 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:50.078386 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:58:50.078450 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:50.078414 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:58:50.486828 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:50.486777 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" event={"ID":"aacd798b-5303-4259-9536-c4e3e1f542e6","Type":"ContainerStarted","Data":"46975d6b88e60b3b3c254922def6f8fff163af38a134f27dbc612c004e9125db"} Jul 13 14:58:50.487011 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:50.486852 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" event={"ID":"aacd798b-5303-4259-9536-c4e3e1f542e6","Type":"ContainerStarted","Data":"73aa6f6ef2bccca58889ce01b5209806639b0fff27a0811cb1d7873d901a58b3"} Jul 13 14:58:50.508121 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:50.508074 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" podStartSLOduration=1.5080594619999999 podStartE2EDuration="1.508059462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:58:50.506558449 +0000 UTC m=+2272.190978755" watchObservedRunningTime="2026-07-13 14:58:50.508059462 +0000 UTC m=+2272.192479766" Jul 13 14:58:50.816678 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:50.816645 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:50.924986 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:50.924953 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:51.929698 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:51.929667 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:52.444403 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.444350 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:52.475722 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.475697 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 14:58:52.499311 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.499281 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:52.500295 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.500272 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:58:52.528173 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.528096 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:52.787214 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.787125 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc"] Jul 13 14:58:52.790925 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.790906 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" Jul 13 14:58:52.793324 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.793302 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b132ed-serving-cert\"" Jul 13 14:58:52.800474 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.800451 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc"] Jul 13 14:58:52.817752 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.817727 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:52.849759 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.849724 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnpzj\" (UniqueName: \"kubernetes.io/projected/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-kube-api-access-nnpzj\") pod \"maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc\" (UID: \"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" Jul 13 14:58:52.849910 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.849822 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc\" (UID: \"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" Jul 13 14:58:52.849910 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.849854 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-maas-api-tls\") pod \"maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc\" (UID: \"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" Jul 13 14:58:52.950305 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.950276 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnpzj\" (UniqueName: \"kubernetes.io/projected/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-kube-api-access-nnpzj\") pod \"maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc\" (UID: \"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" Jul 13 14:58:52.950812 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.950324 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc\" (UID: \"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" Jul 13 14:58:52.950812 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.950348 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-maas-api-tls\") pod \"maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc\" (UID: \"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" Jul 13 14:58:52.951116 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.951093 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc\" (UID: \"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" Jul 13 14:58:52.952878 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.952856 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-maas-api-tls\") pod \"maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc\" (UID: \"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" Jul 13 14:58:52.959655 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.959623 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnpzj\" (UniqueName: \"kubernetes.io/projected/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-kube-api-access-nnpzj\") pod \"maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc\" (UID: \"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" Jul 13 14:58:52.989371 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:52.989343 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:53.103563 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:53.103532 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" Jul 13 14:58:53.300981 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:53.300952 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc"] Jul 13 14:58:53.302333 ip-10-0-131-188 kubenswrapper[2566]: W0713 14:58:53.302301 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf123d5_e5f2_4a96_9164_e3ee6ddc7ea4.slice/crio-2cf169e07e1164bc600d61a0a451381bc2be1b8cb089d5c89f58ad8dfc0c45b0 WatchSource:0}: Error finding container 2cf169e07e1164bc600d61a0a451381bc2be1b8cb089d5c89f58ad8dfc0c45b0: Status 404 returned error can't find the container with id 2cf169e07e1164bc600d61a0a451381bc2be1b8cb089d5c89f58ad8dfc0c45b0 Jul 13 14:58:53.354874 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:53.347197 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:53.439713 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:53.439679 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:53.506489 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:53.506437 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" event={"ID":"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4","Type":"ContainerStarted","Data":"2cf169e07e1164bc600d61a0a451381bc2be1b8cb089d5c89f58ad8dfc0c45b0"} Jul 13 14:58:54.513620 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:54.513578 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" event={"ID":"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4","Type":"ContainerStarted","Data":"5bf315bd70eb5531f9d322d47dc72a302eb83249c53bf909a977382aa2ee22f3"} Jul 13 14:58:54.514080 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:54.513822 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" Jul 13 14:58:54.533285 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:54.533227 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" podStartSLOduration=1.995030699 podStartE2EDuration="2.533205424s" totalImagesPullingTime="538.174725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 14:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 14:58:54.530469288 +0000 UTC m=+2276.214889615" watchObservedRunningTime="2026-07-13 14:58:54.533205424 +0000 UTC m=+2276.217625731" Jul 13 14:58:56.957552 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:56.957519 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:58:59.487969 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:59.487925 2566 patch_prober.go:28] interesting pod/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 13 14:58:59.488346 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:58:59.487999 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" podUID="58ea89bf-0800-4402-96f1-945b197af8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 13 14:59:00.523840 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:00.523812 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" Jul 13 14:59:00.559028 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:00.558982 2566 generic.go:356] "Generic (PLEG): container finished" podID="58ea89bf-0800-4402-96f1-945b197af8de" containerID="4a31a17c16347036914bb4eb90fb96fccad26137e46fce0eeb62c5c686881e20" exitCode=0 Jul 13 14:59:00.559229 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:00.559138 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" event={"ID":"58ea89bf-0800-4402-96f1-945b197af8de","Type":"ContainerDied","Data":"4a31a17c16347036914bb4eb90fb96fccad26137e46fce0eeb62c5c686881e20"} Jul 13 14:59:00.559688 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:00.559664 2566 scope.go:117] "RemoveContainer" containerID="4a31a17c16347036914bb4eb90fb96fccad26137e46fce0eeb62c5c686881e20" Jul 13 14:59:00.561647 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:00.561607 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:59:00.561760 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:00.561704 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:59:00.561760 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:00.561753 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:59:00.792331 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:00.792250 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 14:59:01.575252 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:01.575208 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" event={"ID":"58ea89bf-0800-4402-96f1-945b197af8de","Type":"ContainerStarted","Data":"d219575cf4ad7c7e2f3e4dc9251b5c00ca306d393e90f71d0c82e8c75307b342"} Jul 13 14:59:02.146548 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:02.146072 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:59:02.146548 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:02.146206 2566 patch_prober.go:28] interesting pod/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 13 14:59:02.146548 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:02.146261 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" podUID="58ea89bf-0800-4402-96f1-945b197af8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 13 14:59:03.146107 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:03.146061 2566 patch_prober.go:28] interesting pod/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 13 14:59:03.146617 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:03.146130 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" podUID="58ea89bf-0800-4402-96f1-945b197af8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 13 14:59:04.146204 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:04.146158 2566 patch_prober.go:28] interesting pod/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 13 14:59:04.146682 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:04.146252 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" podUID="58ea89bf-0800-4402-96f1-945b197af8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 13 14:59:05.146026 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:05.145989 2566 patch_prober.go:28] interesting pod/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Jul 13 14:59:05.146333 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:05.146307 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" podUID="58ea89bf-0800-4402-96f1-945b197af8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Jul 13 14:59:06.149773 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:06.149740 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:59:06.150164 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:06.150149 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:59:06.150716 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:06.150699 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 14:59:07.500851 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:07.500820 2566 patch_prober.go:28] interesting pod/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 13 14:59:07.501211 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:07.500876 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" podUID="aacd798b-5303-4259-9536-c4e3e1f542e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 13 14:59:07.612355 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:07.612284 2566 generic.go:356] "Generic (PLEG): container finished" podID="aacd798b-5303-4259-9536-c4e3e1f542e6" containerID="46975d6b88e60b3b3c254922def6f8fff163af38a134f27dbc612c004e9125db" exitCode=0 Jul 13 14:59:07.612492 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:07.612355 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" event={"ID":"aacd798b-5303-4259-9536-c4e3e1f542e6","Type":"ContainerDied","Data":"46975d6b88e60b3b3c254922def6f8fff163af38a134f27dbc612c004e9125db"} Jul 13 14:59:07.612918 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:07.612903 2566 scope.go:117] "RemoveContainer" containerID="46975d6b88e60b3b3c254922def6f8fff163af38a134f27dbc612c004e9125db" Jul 13 14:59:07.613851 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:07.613826 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:59:07.613963 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:07.613902 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:59:07.613963 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:07.613948 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 14:59:08.619665 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:08.619626 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" event={"ID":"aacd798b-5303-4259-9536-c4e3e1f542e6","Type":"ContainerStarted","Data":"5e92280e521202fc11fa0129ec838260406c8b91ac396fce545e765f4f554860"} Jul 13 14:59:08.924492 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:08.924406 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:59:08.924642 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:08.924577 2566 patch_prober.go:28] interesting pod/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 13 14:59:08.924642 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:08.924618 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" podUID="aacd798b-5303-4259-9536-c4e3e1f542e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 13 14:59:09.924746 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:09.924708 2566 patch_prober.go:28] interesting pod/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 13 14:59:09.925139 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:09.924762 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" podUID="aacd798b-5303-4259-9536-c4e3e1f542e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 13 14:59:10.925147 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:10.925107 2566 patch_prober.go:28] interesting pod/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 13 14:59:10.925561 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:10.925177 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" podUID="aacd798b-5303-4259-9536-c4e3e1f542e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 13 14:59:11.924975 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:11.924937 2566 patch_prober.go:28] interesting pod/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Jul 13 14:59:11.925154 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:11.925003 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" podUID="aacd798b-5303-4259-9536-c4e3e1f542e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Jul 13 14:59:12.928916 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:12.928870 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:59:12.929363 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:12.929313 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:59:12.930446 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:12.930424 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 14:59:36.726360 ip-10-0-131-188 kubenswrapper[2566]: I0713 14:59:36.726316 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:05.968746 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:05.968635 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:06.915727 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:06.915686 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:07.019337 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.019299 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz"] Jul 13 15:00:07.019802 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.019517 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" podUID="8f01234c-ec08-4a24-8fe7-41ab4198268b" containerName="maas-api" containerID="cri-o://23531a5aa6ee814563376a3bb7a318cfd759e3692cd9f841caea7c1f84d2213d" gracePeriod=30 Jul 13 15:00:07.056057 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.056030 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:07.118572 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.118537 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:07.275132 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.275106 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 15:00:07.382814 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.382754 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6rzz\" (UniqueName: \"kubernetes.io/projected/8f01234c-ec08-4a24-8fe7-41ab4198268b-kube-api-access-w6rzz\") pod \"8f01234c-ec08-4a24-8fe7-41ab4198268b\" (UID: \"8f01234c-ec08-4a24-8fe7-41ab4198268b\") " Jul 13 15:00:07.383007 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.382856 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f01234c-ec08-4a24-8fe7-41ab4198268b-openshift-service-ca\") pod \"8f01234c-ec08-4a24-8fe7-41ab4198268b\" (UID: \"8f01234c-ec08-4a24-8fe7-41ab4198268b\") " Jul 13 15:00:07.383007 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.382879 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f01234c-ec08-4a24-8fe7-41ab4198268b-maas-api-tls\") pod \"8f01234c-ec08-4a24-8fe7-41ab4198268b\" (UID: \"8f01234c-ec08-4a24-8fe7-41ab4198268b\") " Jul 13 15:00:07.383198 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.383170 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f01234c-ec08-4a24-8fe7-41ab4198268b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f01234c-ec08-4a24-8fe7-41ab4198268b" (UID: "8f01234c-ec08-4a24-8fe7-41ab4198268b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:00:07.384959 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.384924 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f01234c-ec08-4a24-8fe7-41ab4198268b-kube-api-access-w6rzz" (OuterVolumeSpecName: "kube-api-access-w6rzz") pod "8f01234c-ec08-4a24-8fe7-41ab4198268b" (UID: "8f01234c-ec08-4a24-8fe7-41ab4198268b"). InnerVolumeSpecName "kube-api-access-w6rzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:00:07.384959 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.384922 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f01234c-ec08-4a24-8fe7-41ab4198268b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f01234c-ec08-4a24-8fe7-41ab4198268b" (UID: "8f01234c-ec08-4a24-8fe7-41ab4198268b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 15:00:07.478280 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.478200 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:07.484257 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.484235 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6rzz\" (UniqueName: \"kubernetes.io/projected/8f01234c-ec08-4a24-8fe7-41ab4198268b-kube-api-access-w6rzz\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:00:07.484321 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.484260 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f01234c-ec08-4a24-8fe7-41ab4198268b-openshift-service-ca\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:00:07.484321 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.484271 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f01234c-ec08-4a24-8fe7-41ab4198268b-maas-api-tls\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:00:07.954464 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.954432 2566 generic.go:356] "Generic (PLEG): container finished" podID="8f01234c-ec08-4a24-8fe7-41ab4198268b" containerID="23531a5aa6ee814563376a3bb7a318cfd759e3692cd9f841caea7c1f84d2213d" exitCode=0 Jul 13 15:00:07.954631 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.954481 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" Jul 13 15:00:07.954631 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.954481 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" event={"ID":"8f01234c-ec08-4a24-8fe7-41ab4198268b","Type":"ContainerDied","Data":"23531a5aa6ee814563376a3bb7a318cfd759e3692cd9f841caea7c1f84d2213d"} Jul 13 15:00:07.954631 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.954607 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz" event={"ID":"8f01234c-ec08-4a24-8fe7-41ab4198268b","Type":"ContainerDied","Data":"f93e675b72fc0d7acd40f1c4fa3140e340aee36f9b31400093ec37034b9dd596"} Jul 13 15:00:07.954768 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.954631 2566 scope.go:117] "RemoveContainer" containerID="23531a5aa6ee814563376a3bb7a318cfd759e3692cd9f841caea7c1f84d2213d" Jul 13 15:00:07.964358 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.964343 2566 scope.go:117] "RemoveContainer" containerID="23531a5aa6ee814563376a3bb7a318cfd759e3692cd9f841caea7c1f84d2213d" Jul 13 15:00:07.964604 ip-10-0-131-188 kubenswrapper[2566]: E0713 15:00:07.964586 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23531a5aa6ee814563376a3bb7a318cfd759e3692cd9f841caea7c1f84d2213d\": container with ID starting with 23531a5aa6ee814563376a3bb7a318cfd759e3692cd9f841caea7c1f84d2213d not found: ID does not exist" containerID="23531a5aa6ee814563376a3bb7a318cfd759e3692cd9f841caea7c1f84d2213d" Jul 13 15:00:07.964652 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.964613 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23531a5aa6ee814563376a3bb7a318cfd759e3692cd9f841caea7c1f84d2213d"} err="failed to get container status \"23531a5aa6ee814563376a3bb7a318cfd759e3692cd9f841caea7c1f84d2213d\": rpc error: code = NotFound desc = could not find container \"23531a5aa6ee814563376a3bb7a318cfd759e3692cd9f841caea7c1f84d2213d\": container with ID starting with 23531a5aa6ee814563376a3bb7a318cfd759e3692cd9f841caea7c1f84d2213d not found: ID does not exist" Jul 13 15:00:07.977667 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.977646 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz"] Jul 13 15:00:07.981859 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:07.981838 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d6591f-6cc759488d-mjgnz"] Jul 13 15:00:08.869185 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:08.869151 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f01234c-ec08-4a24-8fe7-41ab4198268b" path="/var/lib/kubelet/pods/8f01234c-ec08-4a24-8fe7-41ab4198268b/volumes" Jul 13 15:00:12.647240 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:12.647201 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:43.229495 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:43.229453 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:50.139810 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:50.139758 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:50.414849 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:50.414749 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6"] Jul 13 15:00:50.415002 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:50.414968 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" podUID="58ea89bf-0800-4402-96f1-945b197af8de" containerName="istio-proxy" containerID="cri-o://d219575cf4ad7c7e2f3e4dc9251b5c00ca306d393e90f71d0c82e8c75307b342" gracePeriod=30 Jul 13 15:00:50.431893 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:50.431858 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:50.708764 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:50.708676 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:50.838876 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:50.838839 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc"] Jul 13 15:00:50.839297 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:50.839116 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" podUID="0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4" containerName="maas-api" containerID="cri-o://5bf315bd70eb5531f9d322d47dc72a302eb83249c53bf909a977382aa2ee22f3" gracePeriod=30 Jul 13 15:00:50.871431 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:50.871403 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:50.930733 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:50.930693 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:51.093506 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.093483 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" Jul 13 15:00:51.151913 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.151881 2566 patch_prober.go:28] interesting pod/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 13 15:00:51.152263 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.151941 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" podUID="58ea89bf-0800-4402-96f1-945b197af8de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 13 15:00:51.183899 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.183868 2566 generic.go:356] "Generic (PLEG): container finished" podID="0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4" containerID="5bf315bd70eb5531f9d322d47dc72a302eb83249c53bf909a977382aa2ee22f3" exitCode=0 Jul 13 15:00:51.184053 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.183903 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" event={"ID":"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4","Type":"ContainerDied","Data":"5bf315bd70eb5531f9d322d47dc72a302eb83249c53bf909a977382aa2ee22f3"} Jul 13 15:00:51.184053 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.183914 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" Jul 13 15:00:51.184053 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.183939 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc" event={"ID":"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4","Type":"ContainerDied","Data":"2cf169e07e1164bc600d61a0a451381bc2be1b8cb089d5c89f58ad8dfc0c45b0"} Jul 13 15:00:51.184053 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.183961 2566 scope.go:117] "RemoveContainer" containerID="5bf315bd70eb5531f9d322d47dc72a302eb83249c53bf909a977382aa2ee22f3" Jul 13 15:00:51.193014 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.192978 2566 scope.go:117] "RemoveContainer" containerID="5bf315bd70eb5531f9d322d47dc72a302eb83249c53bf909a977382aa2ee22f3" Jul 13 15:00:51.193277 ip-10-0-131-188 kubenswrapper[2566]: E0713 15:00:51.193241 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf315bd70eb5531f9d322d47dc72a302eb83249c53bf909a977382aa2ee22f3\": container with ID starting with 5bf315bd70eb5531f9d322d47dc72a302eb83249c53bf909a977382aa2ee22f3 not found: ID does not exist" containerID="5bf315bd70eb5531f9d322d47dc72a302eb83249c53bf909a977382aa2ee22f3" Jul 13 15:00:51.193398 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.193284 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf315bd70eb5531f9d322d47dc72a302eb83249c53bf909a977382aa2ee22f3"} err="failed to get container status \"5bf315bd70eb5531f9d322d47dc72a302eb83249c53bf909a977382aa2ee22f3\": rpc error: code = NotFound desc = could not find container \"5bf315bd70eb5531f9d322d47dc72a302eb83249c53bf909a977382aa2ee22f3\": container with ID starting with 5bf315bd70eb5531f9d322d47dc72a302eb83249c53bf909a977382aa2ee22f3 not found: ID does not exist" Jul 13 15:00:51.194572 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.194553 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnpzj\" (UniqueName: \"kubernetes.io/projected/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-kube-api-access-nnpzj\") pod \"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4\" (UID: \"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4\") " Jul 13 15:00:51.194664 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.194582 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-maas-api-tls\") pod \"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4\" (UID: \"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4\") " Jul 13 15:00:51.194664 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.194638 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-openshift-service-ca\") pod \"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4\" (UID: \"0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4\") " Jul 13 15:00:51.195014 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.194994 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4" (UID: "0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:00:51.196593 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.196563 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4" (UID: "0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 15:00:51.196684 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.196569 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-kube-api-access-nnpzj" (OuterVolumeSpecName: "kube-api-access-nnpzj") pod "0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4" (UID: "0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4"). InnerVolumeSpecName "kube-api-access-nnpzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:00:51.295917 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.295841 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnpzj\" (UniqueName: \"kubernetes.io/projected/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-kube-api-access-nnpzj\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:00:51.295917 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.295866 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-maas-api-tls\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:00:51.295917 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.295877 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4-openshift-service-ca\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:00:51.321371 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.321335 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:51.510800 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.510763 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc"] Jul 13 15:00:51.516533 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:51.516505 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b132ed-b8f9846b7-qrjqc"] Jul 13 15:00:52.867985 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:52.867950 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4" path="/var/lib/kubelet/pods/0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4/volumes" Jul 13 15:00:55.672235 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.672212 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 15:00:55.735545 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.735462 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-workload-certs\") pod \"58ea89bf-0800-4402-96f1-945b197af8de\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " Jul 13 15:00:55.735545 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.735527 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ea89bf-0800-4402-96f1-945b197af8de-istiod-ca-cert\") pod \"58ea89bf-0800-4402-96f1-945b197af8de\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " Jul 13 15:00:55.735778 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.735555 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-workload-socket\") pod \"58ea89bf-0800-4402-96f1-945b197af8de\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " Jul 13 15:00:55.735778 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.735577 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-istio-data\") pod \"58ea89bf-0800-4402-96f1-945b197af8de\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " Jul 13 15:00:55.735778 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.735627 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-istio-envoy\") pod \"58ea89bf-0800-4402-96f1-945b197af8de\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " Jul 13 15:00:55.735778 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.735652 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ea89bf-0800-4402-96f1-945b197af8de-istio-token\") pod \"58ea89bf-0800-4402-96f1-945b197af8de\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " Jul 13 15:00:55.735778 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.735678 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-credential-socket\") pod \"58ea89bf-0800-4402-96f1-945b197af8de\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " Jul 13 15:00:55.735778 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.735712 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxc9r\" (UniqueName: \"kubernetes.io/projected/58ea89bf-0800-4402-96f1-945b197af8de-kube-api-access-pxc9r\") pod \"58ea89bf-0800-4402-96f1-945b197af8de\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " Jul 13 15:00:55.735778 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.735738 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ea89bf-0800-4402-96f1-945b197af8de-istio-podinfo\") pod \"58ea89bf-0800-4402-96f1-945b197af8de\" (UID: \"58ea89bf-0800-4402-96f1-945b197af8de\") " Jul 13 15:00:55.736171 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.735806 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "58ea89bf-0800-4402-96f1-945b197af8de" (UID: "58ea89bf-0800-4402-96f1-945b197af8de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:00:55.736171 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.735973 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "58ea89bf-0800-4402-96f1-945b197af8de" (UID: "58ea89bf-0800-4402-96f1-945b197af8de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:00:55.736171 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.736110 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-workload-certs\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:00:55.736171 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.736130 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-credential-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:00:55.736171 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.736155 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ea89bf-0800-4402-96f1-945b197af8de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "58ea89bf-0800-4402-96f1-945b197af8de" (UID: "58ea89bf-0800-4402-96f1-945b197af8de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:00:55.736415 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.736322 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "58ea89bf-0800-4402-96f1-945b197af8de" (UID: "58ea89bf-0800-4402-96f1-945b197af8de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:00:55.736770 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.736743 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-istio-data" (OuterVolumeSpecName: "istio-data") pod "58ea89bf-0800-4402-96f1-945b197af8de" (UID: "58ea89bf-0800-4402-96f1-945b197af8de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:00:55.738213 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.738184 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "58ea89bf-0800-4402-96f1-945b197af8de" (UID: "58ea89bf-0800-4402-96f1-945b197af8de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:00:55.738301 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.738217 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ea89bf-0800-4402-96f1-945b197af8de-kube-api-access-pxc9r" (OuterVolumeSpecName: "kube-api-access-pxc9r") pod "58ea89bf-0800-4402-96f1-945b197af8de" (UID: "58ea89bf-0800-4402-96f1-945b197af8de"). InnerVolumeSpecName "kube-api-access-pxc9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:00:55.738470 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.738448 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ea89bf-0800-4402-96f1-945b197af8de-istio-token" (OuterVolumeSpecName: "istio-token") pod "58ea89bf-0800-4402-96f1-945b197af8de" (UID: "58ea89bf-0800-4402-96f1-945b197af8de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:00:55.738508 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.738467 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58ea89bf-0800-4402-96f1-945b197af8de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "58ea89bf-0800-4402-96f1-945b197af8de" (UID: "58ea89bf-0800-4402-96f1-945b197af8de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:00:55.836983 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.836946 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxc9r\" (UniqueName: \"kubernetes.io/projected/58ea89bf-0800-4402-96f1-945b197af8de-kube-api-access-pxc9r\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:00:55.836983 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.836976 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ea89bf-0800-4402-96f1-945b197af8de-istio-podinfo\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:00:55.836983 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.836985 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ea89bf-0800-4402-96f1-945b197af8de-istiod-ca-cert\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:00:55.836983 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.836993 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-workload-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:00:55.836983 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.837003 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-istio-data\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:00:55.837298 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.837011 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ea89bf-0800-4402-96f1-945b197af8de-istio-envoy\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:00:55.837298 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:55.837019 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ea89bf-0800-4402-96f1-945b197af8de-istio-token\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:00:56.211893 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:56.211852 2566 generic.go:356] "Generic (PLEG): container finished" podID="58ea89bf-0800-4402-96f1-945b197af8de" containerID="d219575cf4ad7c7e2f3e4dc9251b5c00ca306d393e90f71d0c82e8c75307b342" exitCode=0 Jul 13 15:00:56.211893 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:56.211896 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" event={"ID":"58ea89bf-0800-4402-96f1-945b197af8de","Type":"ContainerDied","Data":"d219575cf4ad7c7e2f3e4dc9251b5c00ca306d393e90f71d0c82e8c75307b342"} Jul 13 15:00:56.212145 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:56.211905 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" Jul 13 15:00:56.212145 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:56.211945 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6" event={"ID":"58ea89bf-0800-4402-96f1-945b197af8de","Type":"ContainerDied","Data":"d2b0d58ee2b30a74969c718040bd903056781389ae867e44a98994fa09bf88a3"} Jul 13 15:00:56.212145 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:56.211969 2566 scope.go:117] "RemoveContainer" containerID="d219575cf4ad7c7e2f3e4dc9251b5c00ca306d393e90f71d0c82e8c75307b342" Jul 13 15:00:56.222095 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:56.222072 2566 scope.go:117] "RemoveContainer" containerID="4a31a17c16347036914bb4eb90fb96fccad26137e46fce0eeb62c5c686881e20" Jul 13 15:00:56.230220 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:56.230197 2566 scope.go:117] "RemoveContainer" containerID="d219575cf4ad7c7e2f3e4dc9251b5c00ca306d393e90f71d0c82e8c75307b342" Jul 13 15:00:56.230526 ip-10-0-131-188 kubenswrapper[2566]: E0713 15:00:56.230501 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d219575cf4ad7c7e2f3e4dc9251b5c00ca306d393e90f71d0c82e8c75307b342\": container with ID starting with d219575cf4ad7c7e2f3e4dc9251b5c00ca306d393e90f71d0c82e8c75307b342 not found: ID does not exist" containerID="d219575cf4ad7c7e2f3e4dc9251b5c00ca306d393e90f71d0c82e8c75307b342" Jul 13 15:00:56.230619 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:56.230532 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d219575cf4ad7c7e2f3e4dc9251b5c00ca306d393e90f71d0c82e8c75307b342"} err="failed to get container status \"d219575cf4ad7c7e2f3e4dc9251b5c00ca306d393e90f71d0c82e8c75307b342\": rpc error: code = NotFound desc = could not find container \"d219575cf4ad7c7e2f3e4dc9251b5c00ca306d393e90f71d0c82e8c75307b342\": container with ID starting with d219575cf4ad7c7e2f3e4dc9251b5c00ca306d393e90f71d0c82e8c75307b342 not found: ID does not exist" Jul 13 15:00:56.230619 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:56.230551 2566 scope.go:117] "RemoveContainer" containerID="4a31a17c16347036914bb4eb90fb96fccad26137e46fce0eeb62c5c686881e20" Jul 13 15:00:56.230851 ip-10-0-131-188 kubenswrapper[2566]: E0713 15:00:56.230832 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a31a17c16347036914bb4eb90fb96fccad26137e46fce0eeb62c5c686881e20\": container with ID starting with 4a31a17c16347036914bb4eb90fb96fccad26137e46fce0eeb62c5c686881e20 not found: ID does not exist" containerID="4a31a17c16347036914bb4eb90fb96fccad26137e46fce0eeb62c5c686881e20" Jul 13 15:00:56.230899 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:56.230859 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a31a17c16347036914bb4eb90fb96fccad26137e46fce0eeb62c5c686881e20"} err="failed to get container status \"4a31a17c16347036914bb4eb90fb96fccad26137e46fce0eeb62c5c686881e20\": rpc error: code = NotFound desc = could not find container \"4a31a17c16347036914bb4eb90fb96fccad26137e46fce0eeb62c5c686881e20\": container with ID starting with 4a31a17c16347036914bb4eb90fb96fccad26137e46fce0eeb62c5c686881e20 not found: ID does not exist" Jul 13 15:00:56.236188 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:56.236154 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6"] Jul 13 15:00:56.242015 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:56.241992 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d6591f-openshift-default-75bc86d5d7-zzbh6"] Jul 13 15:00:56.867858 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:56.867825 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58ea89bf-0800-4402-96f1-945b197af8de" path="/var/lib/kubelet/pods/58ea89bf-0800-4402-96f1-945b197af8de/volumes" Jul 13 15:00:57.784136 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:57.784102 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:57.883016 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:57.882984 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:58.039034 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.038952 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb"] Jul 13 15:00:58.039250 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.039224 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" podUID="aacd798b-5303-4259-9536-c4e3e1f542e6" containerName="istio-proxy" containerID="cri-o://5e92280e521202fc11fa0129ec838260406c8b91ac396fce545e765f4f554860" gracePeriod=30 Jul 13 15:00:58.053595 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.053568 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:58.873045 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.873001 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:58.966087 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.966051 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:58.974759 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.974735 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf"] Jul 13 15:00:58.975294 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.975266 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ea89bf-0800-4402-96f1-945b197af8de" containerName="istio-proxy" Jul 13 15:00:58.975294 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.975293 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ea89bf-0800-4402-96f1-945b197af8de" containerName="istio-proxy" Jul 13 15:00:58.975481 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.975314 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f01234c-ec08-4a24-8fe7-41ab4198268b" containerName="maas-api" Jul 13 15:00:58.975481 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.975321 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f01234c-ec08-4a24-8fe7-41ab4198268b" containerName="maas-api" Jul 13 15:00:58.975481 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.975345 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4" containerName="maas-api" Jul 13 15:00:58.975481 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.975351 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4" containerName="maas-api" Jul 13 15:00:58.975481 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.975434 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ea89bf-0800-4402-96f1-945b197af8de" containerName="istio-proxy" Jul 13 15:00:58.975481 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.975445 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f01234c-ec08-4a24-8fe7-41ab4198268b" containerName="maas-api" Jul 13 15:00:58.975481 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.975454 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf123d5-e5f2-4a96-9164-e3ee6ddc7ea4" containerName="maas-api" Jul 13 15:00:58.980092 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.980073 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:58.982927 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.982906 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-93710c-openshift-default-dockercfg-zmmt2\"" Jul 13 15:00:58.990610 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:58.990585 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf"] Jul 13 15:00:59.037049 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.037011 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:00:59.068400 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.068365 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-data\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.068622 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.068409 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-credential-socket\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.068622 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.068448 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istiod-ca-cert\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.068622 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.068469 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzmj\" (UniqueName: \"kubernetes.io/projected/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-kube-api-access-5wzmj\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.068622 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.068518 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-podinfo\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.068622 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.068596 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-workload-certs\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.068869 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.068646 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-workload-socket\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.068869 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.068676 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-envoy\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.068869 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.068737 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-token\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.169367 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.169337 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-token\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.169543 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.169386 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-data\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.169543 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.169415 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-credential-socket\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.169543 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.169448 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istiod-ca-cert\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.169543 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.169470 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzmj\" (UniqueName: \"kubernetes.io/projected/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-kube-api-access-5wzmj\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.169780 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.169628 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-podinfo\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.169780 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.169697 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-workload-certs\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.169780 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.169750 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-workload-socket\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.169988 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.169815 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-envoy\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.169988 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.169818 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-data\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.169988 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.169859 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-credential-socket\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.170099 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.170041 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-workload-certs\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.170099 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.170077 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-workload-socket\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.170208 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.170191 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istiod-ca-cert\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.172042 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.172017 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-envoy\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.172140 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.172117 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-podinfo\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.179613 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.179594 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-token\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.180021 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.180001 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzmj\" (UniqueName: \"kubernetes.io/projected/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-kube-api-access-5wzmj\") pod \"e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.292734 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.292701 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:00:59.442041 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.441948 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf"] Jul 13 15:00:59.444548 ip-10-0-131-188 kubenswrapper[2566]: W0713 15:00:59.444521 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9ee2fc_ec0a_4134_a183_2f0e2c70f1ab.slice/crio-974bdc2e150a1a8c432e3d31bfba75ea2335943a1a8c2c1f1b74814cbf0015fd WatchSource:0}: Error finding container 974bdc2e150a1a8c432e3d31bfba75ea2335943a1a8c2c1f1b74814cbf0015fd: Status 404 returned error can't find the container with id 974bdc2e150a1a8c432e3d31bfba75ea2335943a1a8c2c1f1b74814cbf0015fd Jul 13 15:00:59.446505 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.446474 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 15:00:59.446602 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.446541 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 15:00:59.446602 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.446570 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Jul 13 15:00:59.981219 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:00:59.981184 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:01:00.237007 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:00.236916 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" event={"ID":"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab","Type":"ContainerStarted","Data":"5ec7fc9edf7b7ab9ea5d7a29af622d1879c5cfea5d4ec533a2233e363e58907b"} Jul 13 15:01:00.237007 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:00.236982 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" event={"ID":"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab","Type":"ContainerStarted","Data":"974bdc2e150a1a8c432e3d31bfba75ea2335943a1a8c2c1f1b74814cbf0015fd"} Jul 13 15:01:00.264064 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:00.264021 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" podStartSLOduration=2.264005598 podStartE2EDuration="2.264005598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 15:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:01:00.261805438 +0000 UTC m=+2401.946225761" watchObservedRunningTime="2026-07-13 15:01:00.264005598 +0000 UTC m=+2401.948425905" Jul 13 15:01:00.293530 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:00.293502 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:01:00.298902 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:00.298877 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:01:01.242184 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:01.242149 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:01:01.243051 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:01.243011 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:01:02.208522 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:02.208482 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:01:02.304515 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:02.304469 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:01:03.255887 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.255843 2566 generic.go:356] "Generic (PLEG): container finished" podID="aacd798b-5303-4259-9536-c4e3e1f542e6" containerID="5e92280e521202fc11fa0129ec838260406c8b91ac396fce545e765f4f554860" exitCode=0 Jul 13 15:01:03.256067 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.255903 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" event={"ID":"aacd798b-5303-4259-9536-c4e3e1f542e6","Type":"ContainerDied","Data":"5e92280e521202fc11fa0129ec838260406c8b91ac396fce545e765f4f554860"} Jul 13 15:01:03.256067 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.255969 2566 scope.go:117] "RemoveContainer" containerID="46975d6b88e60b3b3c254922def6f8fff163af38a134f27dbc612c004e9125db" Jul 13 15:01:03.301452 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.301390 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 15:01:03.408246 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.408207 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-credential-socket\") pod \"aacd798b-5303-4259-9536-c4e3e1f542e6\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " Jul 13 15:01:03.408649 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.408267 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swmb2\" (UniqueName: \"kubernetes.io/projected/aacd798b-5303-4259-9536-c4e3e1f542e6-kube-api-access-swmb2\") pod \"aacd798b-5303-4259-9536-c4e3e1f542e6\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " Jul 13 15:01:03.408649 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.408318 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-envoy\") pod \"aacd798b-5303-4259-9536-c4e3e1f542e6\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " Jul 13 15:01:03.408649 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.408360 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-workload-socket\") pod \"aacd798b-5303-4259-9536-c4e3e1f542e6\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " Jul 13 15:01:03.408649 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.408403 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-podinfo\") pod \"aacd798b-5303-4259-9536-c4e3e1f542e6\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " Jul 13 15:01:03.408649 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.408430 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacd798b-5303-4259-9536-c4e3e1f542e6-istiod-ca-cert\") pod \"aacd798b-5303-4259-9536-c4e3e1f542e6\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " Jul 13 15:01:03.408649 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.408470 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-data\") pod \"aacd798b-5303-4259-9536-c4e3e1f542e6\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " Jul 13 15:01:03.408649 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.408515 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-workload-certs\") pod \"aacd798b-5303-4259-9536-c4e3e1f542e6\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " Jul 13 15:01:03.408649 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.408542 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-token\") pod \"aacd798b-5303-4259-9536-c4e3e1f542e6\" (UID: \"aacd798b-5303-4259-9536-c4e3e1f542e6\") " Jul 13 15:01:03.409090 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.408467 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aacd798b-5303-4259-9536-c4e3e1f542e6" (UID: "aacd798b-5303-4259-9536-c4e3e1f542e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:01:03.409090 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.408710 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aacd798b-5303-4259-9536-c4e3e1f542e6" (UID: "aacd798b-5303-4259-9536-c4e3e1f542e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:01:03.409090 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.408952 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-workload-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:01:03.409090 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.408972 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-credential-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:01:03.409090 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.409022 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aacd798b-5303-4259-9536-c4e3e1f542e6" (UID: "aacd798b-5303-4259-9536-c4e3e1f542e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:01:03.409350 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.409216 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacd798b-5303-4259-9536-c4e3e1f542e6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aacd798b-5303-4259-9536-c4e3e1f542e6" (UID: "aacd798b-5303-4259-9536-c4e3e1f542e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:01:03.409533 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.409512 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-data" (OuterVolumeSpecName: "istio-data") pod "aacd798b-5303-4259-9536-c4e3e1f542e6" (UID: "aacd798b-5303-4259-9536-c4e3e1f542e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:01:03.410810 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.410761 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aacd798b-5303-4259-9536-c4e3e1f542e6" (UID: "aacd798b-5303-4259-9536-c4e3e1f542e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:01:03.411028 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.411001 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-token" (OuterVolumeSpecName: "istio-token") pod "aacd798b-5303-4259-9536-c4e3e1f542e6" (UID: "aacd798b-5303-4259-9536-c4e3e1f542e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:01:03.411098 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.411017 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacd798b-5303-4259-9536-c4e3e1f542e6-kube-api-access-swmb2" (OuterVolumeSpecName: "kube-api-access-swmb2") pod "aacd798b-5303-4259-9536-c4e3e1f542e6" (UID: "aacd798b-5303-4259-9536-c4e3e1f542e6"). InnerVolumeSpecName "kube-api-access-swmb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:01:03.411098 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.411011 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aacd798b-5303-4259-9536-c4e3e1f542e6" (UID: "aacd798b-5303-4259-9536-c4e3e1f542e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:01:03.510120 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.510086 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-envoy\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:01:03.510120 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.510114 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-podinfo\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:01:03.510120 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.510126 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacd798b-5303-4259-9536-c4e3e1f542e6-istiod-ca-cert\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:01:03.510344 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.510137 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-data\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:01:03.510344 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.510148 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacd798b-5303-4259-9536-c4e3e1f542e6-workload-certs\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:01:03.510344 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.510158 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacd798b-5303-4259-9536-c4e3e1f542e6-istio-token\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:01:03.510344 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:03.510171 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swmb2\" (UniqueName: \"kubernetes.io/projected/aacd798b-5303-4259-9536-c4e3e1f542e6-kube-api-access-swmb2\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:01:04.262316 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:04.262283 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" Jul 13 15:01:04.262523 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:04.262282 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb" event={"ID":"aacd798b-5303-4259-9536-c4e3e1f542e6","Type":"ContainerDied","Data":"73aa6f6ef2bccca58889ce01b5209806639b0fff27a0811cb1d7873d901a58b3"} Jul 13 15:01:04.262523 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:04.262434 2566 scope.go:117] "RemoveContainer" containerID="5e92280e521202fc11fa0129ec838260406c8b91ac396fce545e765f4f554860" Jul 13 15:01:04.286899 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:04.286872 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb"] Jul 13 15:01:04.294600 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:04.294577 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b132ed-openshift-default-74cb6489c-stfgb"] Jul 13 15:01:04.868918 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:04.868888 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aacd798b-5303-4259-9536-c4e3e1f542e6" path="/var/lib/kubelet/pods/aacd798b-5303-4259-9536-c4e3e1f542e6/volumes" Jul 13 15:01:06.706203 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:01:06.706167 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:02:04.475421 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:02:04.475386 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:03:13.904909 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:03:13.904874 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:03:24.419413 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:03:24.419376 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:03:31.977820 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:03:31.977765 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:04:11.606373 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:11.606334 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:04:13.006108 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:13.006076 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:04:18.477700 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:18.477657 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:04:18.577856 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:18.577820 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:04:18.735260 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:18.735182 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf"] Jul 13 15:04:18.735438 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:18.735412 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" podUID="0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab" containerName="istio-proxy" containerID="cri-o://5ec7fc9edf7b7ab9ea5d7a29af622d1879c5cfea5d4ec533a2233e363e58907b" gracePeriod=30 Jul 13 15:04:18.778499 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:18.778464 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:04:19.029933 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:19.029846 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:04:23.991367 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:23.991340 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:04:24.044829 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.044733 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-data\") pod \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " Jul 13 15:04:24.044829 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.044767 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-envoy\") pod \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " Jul 13 15:04:24.045055 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.044842 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istiod-ca-cert\") pod \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " Jul 13 15:04:24.045055 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.044869 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wzmj\" (UniqueName: \"kubernetes.io/projected/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-kube-api-access-5wzmj\") pod \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " Jul 13 15:04:24.045055 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.044890 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-workload-socket\") pod \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " Jul 13 15:04:24.045055 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.044929 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-credential-socket\") pod \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " Jul 13 15:04:24.045055 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.044960 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-workload-certs\") pod \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " Jul 13 15:04:24.045055 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.045012 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-podinfo\") pod \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " Jul 13 15:04:24.045055 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.045035 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-token\") pod \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\" (UID: \"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab\") " Jul 13 15:04:24.045403 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.045047 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-data" (OuterVolumeSpecName: "istio-data") pod "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab" (UID: "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:04:24.045403 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.045241 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab" (UID: "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:04:24.045403 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.045385 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-data\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:04:24.045403 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.045386 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab" (UID: "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:04:24.045403 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.045311 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab" (UID: "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 15:04:24.045668 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.045404 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-workload-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:04:24.045732 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.045677 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab" (UID: "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:04:24.047081 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.047058 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab" (UID: "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 15:04:24.047220 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.047144 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab" (UID: "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 13 15:04:24.047475 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.047454 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-kube-api-access-5wzmj" (OuterVolumeSpecName: "kube-api-access-5wzmj") pod "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab" (UID: "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab"). InnerVolumeSpecName "kube-api-access-5wzmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:04:24.047475 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.047465 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab" (UID: "0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 15:04:24.146300 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.146262 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istiod-ca-cert\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:04:24.146300 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.146295 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wzmj\" (UniqueName: \"kubernetes.io/projected/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-kube-api-access-5wzmj\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:04:24.146300 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.146308 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-credential-socket\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:04:24.146541 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.146320 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-workload-certs\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:04:24.146541 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.146332 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-podinfo\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:04:24.146541 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.146343 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-token\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:04:24.146541 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.146353 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab-istio-envoy\") on node \"ip-10-0-131-188.ec2.internal\" DevicePath \"\"" Jul 13 15:04:24.316979 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.316932 2566 generic.go:356] "Generic (PLEG): container finished" podID="0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab" containerID="5ec7fc9edf7b7ab9ea5d7a29af622d1879c5cfea5d4ec533a2233e363e58907b" exitCode=0 Jul 13 15:04:24.317123 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.316989 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" Jul 13 15:04:24.317123 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.317011 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" event={"ID":"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab","Type":"ContainerDied","Data":"5ec7fc9edf7b7ab9ea5d7a29af622d1879c5cfea5d4ec533a2233e363e58907b"} Jul 13 15:04:24.317123 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.317083 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf" event={"ID":"0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab","Type":"ContainerDied","Data":"974bdc2e150a1a8c432e3d31bfba75ea2335943a1a8c2c1f1b74814cbf0015fd"} Jul 13 15:04:24.317123 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.317106 2566 scope.go:117] "RemoveContainer" containerID="5ec7fc9edf7b7ab9ea5d7a29af622d1879c5cfea5d4ec533a2233e363e58907b" Jul 13 15:04:24.326584 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.326569 2566 scope.go:117] "RemoveContainer" containerID="5ec7fc9edf7b7ab9ea5d7a29af622d1879c5cfea5d4ec533a2233e363e58907b" Jul 13 15:04:24.326863 ip-10-0-131-188 kubenswrapper[2566]: E0713 15:04:24.326840 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec7fc9edf7b7ab9ea5d7a29af622d1879c5cfea5d4ec533a2233e363e58907b\": container with ID starting with 5ec7fc9edf7b7ab9ea5d7a29af622d1879c5cfea5d4ec533a2233e363e58907b not found: ID does not exist" containerID="5ec7fc9edf7b7ab9ea5d7a29af622d1879c5cfea5d4ec533a2233e363e58907b" Jul 13 15:04:24.326972 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.326870 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec7fc9edf7b7ab9ea5d7a29af622d1879c5cfea5d4ec533a2233e363e58907b"} err="failed to get container status \"5ec7fc9edf7b7ab9ea5d7a29af622d1879c5cfea5d4ec533a2233e363e58907b\": rpc error: code = NotFound desc = could not find container \"5ec7fc9edf7b7ab9ea5d7a29af622d1879c5cfea5d4ec533a2233e363e58907b\": container with ID starting with 5ec7fc9edf7b7ab9ea5d7a29af622d1879c5cfea5d4ec533a2233e363e58907b not found: ID does not exist" Jul 13 15:04:24.342441 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.342420 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf"] Jul 13 15:04:24.349019 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.348999 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-93710c-openshift-default-79fcdd958f-kxqpf"] Jul 13 15:04:24.867339 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:24.867309 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab" path="/var/lib/kubelet/pods/0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab/volumes" Jul 13 15:04:26.030197 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:26.030171 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-fmhc2_aeaeaa41-ec74-4ff7-8d1b-3e4288401836/manager/0.log" Jul 13 15:04:26.165727 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:26.165693 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vpm2d_d1e6beff-eb92-4808-aed6-5099f1626ad4/manager/0.log" Jul 13 15:04:26.307499 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:26.307418 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-786f5ff9c9-swdh4_b23f893c-bb7f-4e48-9bfc-d538e2ffadc5/manager/0.log" Jul 13 15:04:26.433993 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:26.433951 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9jkwn_17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb/server/0.log" Jul 13 15:04:26.567203 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:26.567175 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r5gm9_b2395092-7b71-4dcf-9478-cc9676ac3c40/manager/0.log" Jul 13 15:04:26.985768 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:26.985691 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k_9651f48e-4df1-4b58-b29f-a08590d20e04/copy-manifests/0.log" Jul 13 15:04:26.993375 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:26.993341 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k_9651f48e-4df1-4b58-b29f-a08590d20e04/manager/0.log" Jul 13 15:04:27.361873 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:27.361844 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-65ccc6b695-mtzwx_640fbb25-582c-4b19-9564-7c7f365b34f4/maas-api/0.log" Jul 13 15:04:28.306348 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:28.306316 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk_19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e/util/0.log" Jul 13 15:04:28.313841 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:28.313813 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk_19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e/pull/0.log" Jul 13 15:04:28.320855 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:28.320834 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk_19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e/extract/0.log" Jul 13 15:04:28.457319 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:28.457291 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2_8a0f93a6-34cc-4645-b6ce-c2d6480527ed/util/0.log" Jul 13 15:04:28.464592 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:28.464568 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2_8a0f93a6-34cc-4645-b6ce-c2d6480527ed/pull/0.log" Jul 13 15:04:28.471686 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:28.471669 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2_8a0f93a6-34cc-4645-b6ce-c2d6480527ed/extract/0.log" Jul 13 15:04:28.594919 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:28.594895 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw_ff84688b-cb54-437d-b0d3-6204163e4719/util/0.log" Jul 13 15:04:28.602941 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:28.602920 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw_ff84688b-cb54-437d-b0d3-6204163e4719/pull/0.log" Jul 13 15:04:28.610906 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:28.610880 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw_ff84688b-cb54-437d-b0d3-6204163e4719/extract/0.log" Jul 13 15:04:28.732585 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:28.732559 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k_4b56ed62-746d-4647-bdb3-d5f690201e90/util/0.log" Jul 13 15:04:28.740750 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:28.740724 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k_4b56ed62-746d-4647-bdb3-d5f690201e90/pull/0.log" Jul 13 15:04:28.748130 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:28.748112 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k_4b56ed62-746d-4647-bdb3-d5f690201e90/extract/0.log" Jul 13 15:04:29.024194 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:29.024115 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hks7l_0a48aa30-c211-4289-bbc5-edeb0d05d537/manager/0.log" Jul 13 15:04:29.144322 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:29.144288 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-t5p2z_1fb2a287-7049-45b7-b51a-554483e1ee4b/manager/0.log" Jul 13 15:04:29.276903 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:29.276820 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-5f2dr_e453e581-c08e-4391-87d1-af26b5a59d45/kuadrant-console-plugin/0.log" Jul 13 15:04:29.408529 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:29.408500 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-ljg5s_72587244-7f2b-4a79-869e-445dd6ad500a/registry-server/0.log" Jul 13 15:04:29.541251 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:29.541175 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-fpb6n_2f403521-759c-46c6-afee-9d79c8267d01/manager/0.log" Jul 13 15:04:30.172897 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:30.172853 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mhpqk_0410a37e-f431-4f7c-98c6-4b33ccc893fb/istio-proxy/0.log" Jul 13 15:04:30.431174 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:30.431091 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fcc85dc7-crddz_2e8f8592-fe37-438a-8d5a-c9522c404a05/kube-auth-proxy/0.log" Jul 13 15:04:30.688509 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:30.688432 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nsxgv_58626a7f-6711-430f-8a37-ea4262636ae7/istio-proxy/1.log" Jul 13 15:04:30.831692 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:30.831658 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ft98g_a789508e-4ce5-46e1-8af6-f11da38dbea7/payload-pre-processing/0.log" Jul 13 15:04:30.974272 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:30.974184 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lf8ct_95b4c3bb-75c3-4b39-870a-b702580263e7/payload-processing/0.log" Jul 13 15:04:31.467550 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:31.467522 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-gkq4m_6ddd3b17-c725-4c85-8878-44f8515ef818/main/0.log" Jul 13 15:04:31.753544 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:31.753458 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-rrqrz_50e9a061-8bd0-4582-92d7-9cea7b0b0ec4/main/0.log" Jul 13 15:04:32.013650 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:32.013570 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-25c5n_0648a048-bf8d-4f7d-b5a3-237506a491f0/main/0.log" Jul 13 15:04:36.348199 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:36.348163 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:04:36.410251 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:36.410221 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" limit=1024 Jul 13 15:04:39.160091 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:39.160060 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f9wtx_1d1b73e5-52d4-43b9-b2eb-151cb0a5b16b/global-pull-secret-syncer/0.log" Jul 13 15:04:39.262968 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:39.262937 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bmrdc_abb92db6-649f-49f1-8ec1-5f41c88f9cfb/konnectivity-agent/0.log" Jul 13 15:04:39.355904 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:39.355866 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-188.ec2.internal_daa422171c2cfe1891a45971f28fd4ef/haproxy/0.log" Jul 13 15:04:43.916776 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:43.916743 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk_19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e/extract/0.log" Jul 13 15:04:43.940107 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:43.940074 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk_19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e/util/0.log" Jul 13 15:04:43.962814 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:43.962771 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759kmwzk_19ab9bb8-c7c4-49df-8cf7-f9bcaf8e145e/pull/0.log" Jul 13 15:04:43.993071 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:43.993044 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2_8a0f93a6-34cc-4645-b6ce-c2d6480527ed/extract/0.log" Jul 13 15:04:44.017913 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:44.017888 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2_8a0f93a6-34cc-4645-b6ce-c2d6480527ed/util/0.log" Jul 13 15:04:44.043123 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:44.043095 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0llqj2_8a0f93a6-34cc-4645-b6ce-c2d6480527ed/pull/0.log" Jul 13 15:04:44.071693 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:44.071664 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw_ff84688b-cb54-437d-b0d3-6204163e4719/extract/0.log" Jul 13 15:04:44.093323 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:44.093296 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw_ff84688b-cb54-437d-b0d3-6204163e4719/util/0.log" Jul 13 15:04:44.116612 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:44.116586 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73tmmnw_ff84688b-cb54-437d-b0d3-6204163e4719/pull/0.log" Jul 13 15:04:44.144558 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:44.144531 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k_4b56ed62-746d-4647-bdb3-d5f690201e90/extract/0.log" Jul 13 15:04:44.167124 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:44.167053 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k_4b56ed62-746d-4647-bdb3-d5f690201e90/util/0.log" Jul 13 15:04:44.191925 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:44.191895 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1llp4k_4b56ed62-746d-4647-bdb3-d5f690201e90/pull/0.log" Jul 13 15:04:44.586316 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:44.586286 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-hks7l_0a48aa30-c211-4289-bbc5-edeb0d05d537/manager/0.log" Jul 13 15:04:44.608938 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:44.608904 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-t5p2z_1fb2a287-7049-45b7-b51a-554483e1ee4b/manager/0.log" Jul 13 15:04:44.642463 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:44.642435 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-5f2dr_e453e581-c08e-4391-87d1-af26b5a59d45/kuadrant-console-plugin/0.log" Jul 13 15:04:44.694139 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:44.694113 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-ljg5s_72587244-7f2b-4a79-869e-445dd6ad500a/registry-server/0.log" Jul 13 15:04:44.816098 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:44.816072 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-fpb6n_2f403521-759c-46c6-afee-9d79c8267d01/manager/0.log" Jul 13 15:04:47.196724 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:47.196690 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6z662_1a3ae5cf-ea86-49cd-a760-ce305f3bb203/node-exporter/0.log" Jul 13 15:04:47.221737 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:47.221711 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6z662_1a3ae5cf-ea86-49cd-a760-ce305f3bb203/kube-rbac-proxy/0.log" Jul 13 15:04:47.244767 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:47.244745 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6z662_1a3ae5cf-ea86-49cd-a760-ce305f3bb203/init-textfile/0.log" Jul 13 15:04:49.058333 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.058297 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn"] Jul 13 15:04:49.058806 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.058773 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aacd798b-5303-4259-9536-c4e3e1f542e6" containerName="istio-proxy" Jul 13 15:04:49.058882 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.058809 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aacd798b-5303-4259-9536-c4e3e1f542e6" containerName="istio-proxy" Jul 13 15:04:49.058882 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.058822 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab" containerName="istio-proxy" Jul 13 15:04:49.058882 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.058828 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab" containerName="istio-proxy" Jul 13 15:04:49.058882 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.058846 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ea89bf-0800-4402-96f1-945b197af8de" containerName="istio-proxy" Jul 13 15:04:49.058882 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.058853 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ea89bf-0800-4402-96f1-945b197af8de" containerName="istio-proxy" Jul 13 15:04:49.059072 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.058925 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aacd798b-5303-4259-9536-c4e3e1f542e6" containerName="istio-proxy" Jul 13 15:04:49.059072 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.058933 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ea89bf-0800-4402-96f1-945b197af8de" containerName="istio-proxy" Jul 13 15:04:49.059072 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.058940 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aacd798b-5303-4259-9536-c4e3e1f542e6" containerName="istio-proxy" Jul 13 15:04:49.059072 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.058946 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a9ee2fc-ec0a-4134-a183-2f0e2c70f1ab" containerName="istio-proxy" Jul 13 15:04:49.062224 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.062209 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.064941 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.064918 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bbh27\"/\"openshift-service-ca.crt\"" Jul 13 15:04:49.066052 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.066034 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bbh27\"/\"kube-root-ca.crt\"" Jul 13 15:04:49.066120 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.066072 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bbh27\"/\"default-dockercfg-x5lm4\"" Jul 13 15:04:49.072627 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.072600 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn"] Jul 13 15:04:49.166529 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.166493 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c28sf\" (UniqueName: \"kubernetes.io/projected/8040c899-de05-426c-815a-f235d6ea9b6c-kube-api-access-c28sf\") pod \"perf-node-gather-daemonset-j2xgn\" (UID: \"8040c899-de05-426c-815a-f235d6ea9b6c\") " pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.166529 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.166535 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8040c899-de05-426c-815a-f235d6ea9b6c-proc\") pod \"perf-node-gather-daemonset-j2xgn\" (UID: \"8040c899-de05-426c-815a-f235d6ea9b6c\") " pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.166746 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.166597 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8040c899-de05-426c-815a-f235d6ea9b6c-podres\") pod \"perf-node-gather-daemonset-j2xgn\" (UID: \"8040c899-de05-426c-815a-f235d6ea9b6c\") " pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.166746 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.166617 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8040c899-de05-426c-815a-f235d6ea9b6c-lib-modules\") pod \"perf-node-gather-daemonset-j2xgn\" (UID: \"8040c899-de05-426c-815a-f235d6ea9b6c\") " pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.166746 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.166676 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8040c899-de05-426c-815a-f235d6ea9b6c-sys\") pod \"perf-node-gather-daemonset-j2xgn\" (UID: \"8040c899-de05-426c-815a-f235d6ea9b6c\") " pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.267450 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.267417 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8040c899-de05-426c-815a-f235d6ea9b6c-podres\") pod \"perf-node-gather-daemonset-j2xgn\" (UID: \"8040c899-de05-426c-815a-f235d6ea9b6c\") " pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.267450 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.267454 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8040c899-de05-426c-815a-f235d6ea9b6c-lib-modules\") pod \"perf-node-gather-daemonset-j2xgn\" (UID: \"8040c899-de05-426c-815a-f235d6ea9b6c\") " pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.267680 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.267483 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8040c899-de05-426c-815a-f235d6ea9b6c-sys\") pod \"perf-node-gather-daemonset-j2xgn\" (UID: \"8040c899-de05-426c-815a-f235d6ea9b6c\") " pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.267680 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.267549 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c28sf\" (UniqueName: \"kubernetes.io/projected/8040c899-de05-426c-815a-f235d6ea9b6c-kube-api-access-c28sf\") pod \"perf-node-gather-daemonset-j2xgn\" (UID: \"8040c899-de05-426c-815a-f235d6ea9b6c\") " pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.267680 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.267584 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8040c899-de05-426c-815a-f235d6ea9b6c-proc\") pod \"perf-node-gather-daemonset-j2xgn\" (UID: \"8040c899-de05-426c-815a-f235d6ea9b6c\") " pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.267680 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.267600 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8040c899-de05-426c-815a-f235d6ea9b6c-podres\") pod \"perf-node-gather-daemonset-j2xgn\" (UID: \"8040c899-de05-426c-815a-f235d6ea9b6c\") " pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.267680 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.267610 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8040c899-de05-426c-815a-f235d6ea9b6c-lib-modules\") pod \"perf-node-gather-daemonset-j2xgn\" (UID: \"8040c899-de05-426c-815a-f235d6ea9b6c\") " pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.267680 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.267632 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8040c899-de05-426c-815a-f235d6ea9b6c-sys\") pod \"perf-node-gather-daemonset-j2xgn\" (UID: \"8040c899-de05-426c-815a-f235d6ea9b6c\") " pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.267988 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.267685 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8040c899-de05-426c-815a-f235d6ea9b6c-proc\") pod \"perf-node-gather-daemonset-j2xgn\" (UID: \"8040c899-de05-426c-815a-f235d6ea9b6c\") " pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.277718 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.277690 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c28sf\" (UniqueName: \"kubernetes.io/projected/8040c899-de05-426c-815a-f235d6ea9b6c-kube-api-access-c28sf\") pod \"perf-node-gather-daemonset-j2xgn\" (UID: \"8040c899-de05-426c-815a-f235d6ea9b6c\") " pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.372554 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.372516 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:49.508318 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:49.508293 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn"] Jul 13 15:04:49.509692 ip-10-0-131-188 kubenswrapper[2566]: W0713 15:04:49.509662 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8040c899_de05_426c_815a_f235d6ea9b6c.slice/crio-6cca234d3ee191e401c09ddd6023e0bf29d620714d9581eea33483542c3498d1 WatchSource:0}: Error finding container 6cca234d3ee191e401c09ddd6023e0bf29d620714d9581eea33483542c3498d1: Status 404 returned error can't find the container with id 6cca234d3ee191e401c09ddd6023e0bf29d620714d9581eea33483542c3498d1 Jul 13 15:04:50.305713 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:50.305676 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-784fc7cc58-9x5r8_c0aa2379-1c20-46a0-b1ae-0e45b45f2876/console/0.log" Jul 13 15:04:50.453907 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:50.453869 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" event={"ID":"8040c899-de05-426c-815a-f235d6ea9b6c","Type":"ContainerStarted","Data":"26b0e0f773e81c0284ccd4bcec3f2018d2ac213e852d83f774bf05943add5961"} Jul 13 15:04:50.454090 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:50.453930 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" event={"ID":"8040c899-de05-426c-815a-f235d6ea9b6c","Type":"ContainerStarted","Data":"6cca234d3ee191e401c09ddd6023e0bf29d620714d9581eea33483542c3498d1"} Jul 13 15:04:50.454090 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:50.454024 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:50.477736 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:50.477689 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" podStartSLOduration=1.477675135 podStartE2EDuration="1.477675135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 15:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 15:04:50.476604541 +0000 UTC m=+2632.161024847" watchObservedRunningTime="2026-07-13 15:04:50.477675135 +0000 UTC m=+2632.162095440" Jul 13 15:04:52.155505 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:52.155473 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bhvr5_58085733-1f46-4db6-9ae7-1eae8fec5bab/dns/0.log" Jul 13 15:04:52.176875 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:52.176852 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bhvr5_58085733-1f46-4db6-9ae7-1eae8fec5bab/kube-rbac-proxy/0.log" Jul 13 15:04:52.256733 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:52.256707 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xcv69_91eed05b-8ccb-4806-ac18-2ced768c9780/dns-node-resolver/0.log" Jul 13 15:04:52.838276 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:52.838241 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d7bc5cf5c-gwtrd_f7aab2ee-40cf-473b-b318-f1b61ca71070/registry/0.log" Jul 13 15:04:52.863613 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:52.863587 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-76bmg_5962361d-1783-4050-8470-a1b9518399bb/node-ca/0.log" Jul 13 15:04:53.892372 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:53.892341 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mhpqk_0410a37e-f431-4f7c-98c6-4b33ccc893fb/istio-proxy/0.log" Jul 13 15:04:53.992228 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:53.992189 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fcc85dc7-crddz_2e8f8592-fe37-438a-8d5a-c9522c404a05/kube-auth-proxy/0.log" Jul 13 15:04:54.048484 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:54.048453 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nsxgv_58626a7f-6711-430f-8a37-ea4262636ae7/istio-proxy/0.log" Jul 13 15:04:54.104605 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:54.104573 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nsxgv_58626a7f-6711-430f-8a37-ea4262636ae7/istio-proxy/1.log" Jul 13 15:04:54.141866 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:54.141840 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ft98g_a789508e-4ce5-46e1-8af6-f11da38dbea7/payload-pre-processing/0.log" Jul 13 15:04:54.180493 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:54.180423 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lf8ct_95b4c3bb-75c3-4b39-870a-b702580263e7/payload-processing/0.log" Jul 13 15:04:54.909311 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:54.909276 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6dmbf_ea289c9c-fdbc-4830-ad88-6d6fbf43834d/serve-healthcheck-canary/0.log" Jul 13 15:04:55.596251 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:55.596221 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z78ln_bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe/kube-rbac-proxy/0.log" Jul 13 15:04:55.619731 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:55.619702 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z78ln_bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe/exporter/0.log" Jul 13 15:04:55.641806 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:55.641762 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-z78ln_bfeeed1f-6ca0-481d-bd28-e2c4aaf97abe/extractor/0.log" Jul 13 15:04:56.468909 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:56.468882 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bbh27/perf-node-gather-daemonset-j2xgn" Jul 13 15:04:57.948039 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:57.948011 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-fmhc2_aeaeaa41-ec74-4ff7-8d1b-3e4288401836/manager/0.log" Jul 13 15:04:58.113459 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:58.113427 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vpm2d_d1e6beff-eb92-4808-aed6-5099f1626ad4/manager/0.log" Jul 13 15:04:58.247317 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:58.247238 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-786f5ff9c9-swdh4_b23f893c-bb7f-4e48-9bfc-d538e2ffadc5/manager/0.log" Jul 13 15:04:58.268152 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:58.268126 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9jkwn_17e3ff33-cce6-4a3a-896b-f2f1ba02eaeb/server/0.log" Jul 13 15:04:58.373207 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:58.373176 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r5gm9_b2395092-7b71-4dcf-9478-cc9676ac3c40/manager/0.log" Jul 13 15:04:58.583021 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:58.582997 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k_9651f48e-4df1-4b58-b29f-a08590d20e04/manager/0.log" Jul 13 15:04:58.606378 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:58.606358 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67bcd5d8ff-wbw5k_9651f48e-4df1-4b58-b29f-a08590d20e04/copy-manifests/0.log" Jul 13 15:04:59.774357 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:59.774324 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mhpqk_0410a37e-f431-4f7c-98c6-4b33ccc893fb/istio-proxy/0.log" Jul 13 15:04:59.862978 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:59.862945 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fcc85dc7-crddz_2e8f8592-fe37-438a-8d5a-c9522c404a05/kube-auth-proxy/0.log" Jul 13 15:04:59.906262 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:59.906234 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nsxgv_58626a7f-6711-430f-8a37-ea4262636ae7/istio-proxy/0.log" Jul 13 15:04:59.965955 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:59.965926 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nsxgv_58626a7f-6711-430f-8a37-ea4262636ae7/istio-proxy/1.log" Jul 13 15:04:59.994123 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:04:59.994096 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ft98g_a789508e-4ce5-46e1-8af6-f11da38dbea7/payload-pre-processing/0.log" Jul 13 15:05:00.026247 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:05:00.026160 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lf8ct_95b4c3bb-75c3-4b39-870a-b702580263e7/payload-processing/0.log" Jul 13 15:05:01.284341 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:05:01.284311 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4mhpqk_0410a37e-f431-4f7c-98c6-4b33ccc893fb/istio-proxy/0.log" Jul 13 15:05:01.382236 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:05:01.382207 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fcc85dc7-crddz_2e8f8592-fe37-438a-8d5a-c9522c404a05/kube-auth-proxy/0.log" Jul 13 15:05:01.423624 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:05:01.423598 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nsxgv_58626a7f-6711-430f-8a37-ea4262636ae7/istio-proxy/0.log" Jul 13 15:05:01.484379 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:05:01.484346 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-nsxgv_58626a7f-6711-430f-8a37-ea4262636ae7/istio-proxy/1.log" Jul 13 15:05:01.527471 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:05:01.527444 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-ft98g_a789508e-4ce5-46e1-8af6-f11da38dbea7/payload-pre-processing/0.log" Jul 13 15:05:01.565593 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:05:01.565569 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-lf8ct_95b4c3bb-75c3-4b39-870a-b702580263e7/payload-processing/0.log" Jul 13 15:05:05.056776 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:05:05.056686 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-bj582_88a63d66-f350-425c-b173-98c532394025/migrator/0.log" Jul 13 15:05:05.079889 ip-10-0-131-188 kubenswrapper[2566]: I0713 15:05:05.079860 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-bj582_88a63d66-f350-425c-b173-98c532394025/graceful-termination/0.log"