Jul 02 19:00:36.956221 ip-10-0-142-150 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 02 19:00:36.956366 ip-10-0-142-150 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 02 19:00:36.956489 ip-10-0-142-150 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 02 19:00:36.957015 ip-10-0-142-150 systemd[1]: Failed to start Kubernetes Kubelet. Jul 02 19:00:47.008014 ip-10-0-142-150 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 02 19:00:47.008036 ip-10-0-142-150 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 009a94a491bb45639de51074eab32ee0 -- Jul 02 19:02:48.696664 ip-10-0-142-150 systemd[1]: Starting Kubernetes Kubelet... Jul 02 19:02:49.138196 ip-10-0-142-150 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 19:02:49.138196 ip-10-0-142-150 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 02 19:02:49.138196 ip-10-0-142-150 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 19:02:49.138196 ip-10-0-142-150 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 02 19:02:49.138196 ip-10-0-142-150 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 19:02:49.140142 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.140058 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 02 19:02:49.144158 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144139 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:02:49.144158 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144159 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144163 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144167 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144170 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144173 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144177 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144180 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144183 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144186 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144189 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144192 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144198 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144201 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144203 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144206 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144208 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144211 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144214 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144216 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144219 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:02:49.144225 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144221 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144224 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144226 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144229 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144232 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144235 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144237 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144240 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144242 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144245 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144247 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144249 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144253 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144255 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144257 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144260 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144262 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144266 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144269 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144273 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:02:49.144717 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144276 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144278 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144280 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144283 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144285 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144288 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144290 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144293 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144295 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144297 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144300 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144302 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144304 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144307 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144312 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144315 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144317 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144320 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144322 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144325 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:02:49.145199 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144327 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144330 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144333 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144335 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144338 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144360 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144364 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144366 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144370 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144374 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144383 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144385 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144388 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144390 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144393 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144396 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144399 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144401 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144404 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144406 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:02:49.145699 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144409 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144411 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144414 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144416 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144419 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144421 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144426 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144429 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144431 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144434 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144436 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144439 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144441 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144443 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144446 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144448 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144450 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.144453 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145171 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:02:49.146171 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145178 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145181 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145183 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145186 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145188 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145191 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145194 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145196 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145199 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145201 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145205 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145208 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145211 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145215 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145219 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145222 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145224 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145227 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145230 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145234 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:02:49.146635 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145237 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145239 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145241 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145244 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145247 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145249 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145251 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145254 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145256 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145259 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145261 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145264 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145266 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145268 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145271 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145273 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145276 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145279 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145281 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:02:49.147125 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145284 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145286 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145288 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145292 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145295 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145297 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145299 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145302 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145304 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145306 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145310 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145314 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145318 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145320 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145323 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145325 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145328 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145331 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145334 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:02:49.147611 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145336 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145339 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145341 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145343 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145362 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145365 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145368 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145370 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145373 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145376 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145378 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145381 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145384 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145387 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145389 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145392 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145395 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145397 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145400 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145402 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:02:49.148066 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145405 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145407 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145409 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145412 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145416 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145420 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145423 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145425 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145427 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145430 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145433 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145435 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145438 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145442 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145445 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145448 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145451 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145453 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145456 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:02:49.148585 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145459 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145828 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145833 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145836 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145839 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145843 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145845 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145848 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145850 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145853 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145856 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145858 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145860 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145863 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145866 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145868 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145871 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145873 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145878 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145880 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:02:49.149045 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145883 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145885 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145889 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145891 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145894 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145896 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145899 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145901 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145903 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145906 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145908 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145911 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145913 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145916 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145919 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145921 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145924 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145926 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145929 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:02:49.149551 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145931 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145936 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145939 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145942 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145946 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145949 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145952 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145955 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145958 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145960 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145963 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145967 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145969 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145973 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145976 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145979 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145981 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145984 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145987 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:02:49.150044 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145991 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145993 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145996 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.145999 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146001 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146004 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146006 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146011 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146013 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146015 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146018 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146020 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146023 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146025 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146028 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146030 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146033 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146035 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146038 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146040 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:02:49.150522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146042 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146045 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146047 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146050 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146054 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146056 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146059 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146061 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146064 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146066 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146069 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146071 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146073 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146076 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146078 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146080 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146083 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146085 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146088 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146092 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:02:49.150995 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146094 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146097 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146167 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146178 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146185 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146190 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146194 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146197 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146201 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146205 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146208 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146211 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146214 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146217 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146220 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146223 2565 flags.go:64] FLAG: --cgroup-root="" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146227 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146230 2565 flags.go:64] FLAG: --client-ca-file="" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146232 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146235 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146239 2565 flags.go:64] FLAG: --cluster-domain="" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146242 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146244 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146247 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 02 19:02:49.151522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146251 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146254 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146257 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146260 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146263 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146265 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146268 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146272 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146275 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146278 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146282 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146284 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146287 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146290 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146293 2565 flags.go:64] FLAG: --enable-server="true" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146296 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146300 2565 flags.go:64] FLAG: --event-burst="100" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146303 2565 flags.go:64] FLAG: --event-qps="50" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146306 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146309 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146311 2565 flags.go:64] FLAG: --eviction-hard="" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146315 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146318 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146320 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146323 2565 flags.go:64] FLAG: --eviction-soft="" Jul 02 19:02:49.152087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146326 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146329 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146332 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146335 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146338 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146340 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146343 2565 flags.go:64] FLAG: --feature-gates="" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146360 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146363 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146366 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146369 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146372 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146375 2565 flags.go:64] FLAG: --help="false" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146377 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146382 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146385 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146387 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146391 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146394 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146397 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146399 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146402 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146405 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 02 19:02:49.152693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146408 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146411 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146414 2565 flags.go:64] FLAG: --kube-reserved="" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146417 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146420 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146423 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146425 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146428 2565 flags.go:64] FLAG: --lock-file="" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146431 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146434 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146437 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146442 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146444 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146447 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146450 2565 flags.go:64] FLAG: --logging-format="text" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146452 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146455 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146458 2565 flags.go:64] FLAG: --manifest-url="" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146461 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146464 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146467 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146471 2565 flags.go:64] FLAG: --max-pods="110" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146474 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146478 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146481 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 02 19:02:49.153373 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146483 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146486 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146489 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146492 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146499 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146501 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146504 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146507 2565 flags.go:64] FLAG: --pod-cidr="" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146510 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146516 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146519 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146522 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146525 2565 flags.go:64] FLAG: --port="10250" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146528 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146530 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02d6c050dc498db13" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146533 2565 flags.go:64] FLAG: --qos-reserved="" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146536 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146539 2565 flags.go:64] FLAG: --register-node="true" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146541 2565 flags.go:64] FLAG: --register-with-taints="" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146545 2565 flags.go:64] FLAG: --registry-burst="10" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146548 2565 flags.go:64] FLAG: --registry-qps="5" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146550 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146553 2565 flags.go:64] FLAG: --reserved-memory="" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146556 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146559 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 02 19:02:49.153990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146562 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146564 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146567 2565 flags.go:64] FLAG: --runonce="false" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146570 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146573 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146577 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146579 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146582 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146585 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146588 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146591 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146593 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146596 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146599 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146601 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146604 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146607 2565 flags.go:64] FLAG: --system-cgroups="" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146610 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146615 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146617 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146620 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146623 2565 flags.go:64] FLAG: --tls-min-version="" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146626 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146630 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146632 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 02 19:02:49.154611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146635 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146638 2565 flags.go:64] FLAG: --v="2" Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146642 2565 flags.go:64] FLAG: --version="false" Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146646 2565 flags.go:64] FLAG: --vmodule="" Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146650 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.146653 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146763 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146767 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146770 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146772 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146775 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146777 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146781 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146783 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146786 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146788 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146791 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146793 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146795 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146798 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146801 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:02:49.155256 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146803 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146806 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146808 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146811 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146813 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146815 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146818 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146821 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146823 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146827 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146831 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146835 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146837 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146840 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146843 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146846 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146848 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146853 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146856 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:02:49.155787 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146859 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146862 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146865 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146867 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146871 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146873 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146875 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146878 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146881 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146883 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146886 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146888 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146891 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146893 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146896 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146898 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146900 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146904 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146906 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146909 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:02:49.156250 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146911 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146914 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146918 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146920 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146922 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146925 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146927 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146930 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146946 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146950 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146953 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146956 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146958 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146961 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146964 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146967 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146970 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146973 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146976 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146978 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:02:49.156769 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146981 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146983 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146986 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146988 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146991 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146993 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146996 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.146999 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147002 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147005 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147008 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147010 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147012 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147015 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147019 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147021 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147024 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147026 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147029 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147031 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:02:49.157242 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147033 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:02:49.157780 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147036 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:02:49.157780 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147038 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:02:49.157780 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147041 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:02:49.157780 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.147043 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:02:49.157780 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.147865 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 19:02:49.157780 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.154469 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 02 19:02:49.157780 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.154487 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 02 19:02:49.157780 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.154504 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 02 19:02:49.157780 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.154510 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 02 19:02:49.157780 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154577 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:02:49.157780 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154582 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:02:49.157780 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154585 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:02:49.157780 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154588 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:02:49.157780 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154591 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:02:49.157780 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154594 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154596 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154599 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154601 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154604 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154606 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154609 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154612 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154615 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154617 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154620 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154623 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154626 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154628 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154631 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154634 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154636 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154638 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154642 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154644 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:02:49.158178 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154647 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154651 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154654 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154657 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154662 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154665 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154667 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154670 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154673 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154675 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154678 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154680 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154683 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154685 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154687 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154690 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154692 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154695 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154697 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154700 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:02:49.158671 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154702 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154704 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154707 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154709 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154712 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154714 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154718 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154722 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154725 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154728 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154730 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154733 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154735 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154738 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154740 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154743 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154746 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154749 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:02:49.159167 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154753 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154756 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154759 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154762 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154764 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154767 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154770 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154773 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154775 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154778 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154781 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154783 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154785 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154788 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154790 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154793 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154795 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154797 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154800 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154802 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:02:49.159628 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154805 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:02:49.160121 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154807 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:02:49.160121 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154810 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:02:49.160121 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154813 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:02:49.160121 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154815 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:02:49.160121 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154818 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:02:49.160121 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154820 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:02:49.160121 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154823 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:02:49.160121 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154826 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:02:49.160121 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154829 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:02:49.160121 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154832 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:02:49.160121 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154834 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:02:49.160121 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154837 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:02:49.160121 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154840 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:02:49.160121 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154842 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:02:49.160121 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154845 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:02:49.160577 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.154850 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 19:02:49.160577 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154971 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:02:49.160577 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154976 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:02:49.160577 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154979 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:02:49.160577 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154982 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:02:49.160577 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154985 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:02:49.160577 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154988 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:02:49.160577 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154991 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:02:49.160577 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154993 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:02:49.160577 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154995 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:02:49.160577 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.154999 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:02:49.160577 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155003 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:02:49.160577 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155006 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:02:49.160577 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155009 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:02:49.160577 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155011 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155014 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155017 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155019 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155022 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155024 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155027 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155029 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155031 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155034 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155036 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155039 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155042 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155045 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155047 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155050 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155052 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155055 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155057 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155060 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:02:49.160950 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155062 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155065 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155067 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155069 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155072 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155074 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155076 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155079 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155081 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155083 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155086 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155088 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155090 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155093 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155095 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155097 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155100 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155102 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155105 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155107 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:02:49.161457 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155110 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155113 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155115 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155117 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155120 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155122 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155125 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155128 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155132 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155144 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155148 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155151 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155154 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155157 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155160 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155163 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155165 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155167 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155170 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:02:49.161945 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155172 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155175 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155177 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155180 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155182 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155184 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155187 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155189 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155192 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155194 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155197 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155199 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155202 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155204 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155207 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155209 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155211 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155214 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155216 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155218 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:02:49.162406 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155221 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:02:49.162896 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155223 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:02:49.162896 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155226 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:02:49.162896 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155229 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:02:49.162896 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155231 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:02:49.162896 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155234 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:02:49.162896 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:49.155236 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:02:49.162896 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.155241 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 19:02:49.162896 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.155376 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 02 19:02:49.162896 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.157596 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 02 19:02:49.162896 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.158875 2565 server.go:1019] "Starting client certificate rotation" Jul 02 19:02:49.162896 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.159629 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 19:02:49.162896 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.160233 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 19:02:49.165087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.164810 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 19:02:49.166596 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.166577 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 19:02:49.174546 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.174526 2565 log.go:25] "Validated CRI v1 runtime API" Jul 02 19:02:49.176482 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.176464 2565 log.go:25] "Validated CRI v1 image API" Jul 02 19:02:49.178761 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.178747 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 02 19:02:49.181072 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.181047 2565 fs.go:135] Filesystem UUIDs: map[03ecb3ee-7d62-4634-93e4-0d96dfd06156:/dev/nvme0n1p3 2f0d6aac-ea15-4449-98cc-f7071486427f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 02 19:02:49.181114 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.181072 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 02 19:02:49.185277 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.185259 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 19:02:49.187012 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.186898 2565 manager.go:216] Machine: {Timestamp:2026-07-02 19:02:49.1856257 +0000 UTC m=+0.354986897 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099581 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21ae04223ab5a4646759dfa9b38c49 SystemUUID:ec21ae04-223a-b5a4-6467-59dfa9b38c49 BootID:009a94a4-91bb-4563-9de5-1074eab32ee0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f4:cd:2d:bb:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f4:cd:2d:bb:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:1d:8a:04:87:ff Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 02 19:02:49.187012 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.187009 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 02 19:02:49.187120 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.187095 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 02 19:02:49.190458 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.190430 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 02 19:02:49.190599 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.190462 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-150.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 02 19:02:49.191112 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.191102 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 02 19:02:49.191150 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.191115 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 02 19:02:49.191150 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.191123 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 19:02:49.191872 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.191861 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 19:02:49.193919 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.193907 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 02 19:02:49.197216 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.197204 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 02 19:02:49.197329 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.197320 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 02 19:02:49.197398 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.197365 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 02 19:02:49.197398 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.197376 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 02 19:02:49.197398 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.197391 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 02 19:02:49.197398 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.197400 2565 kubelet.go:398] "Adding apiserver pod source" Jul 02 19:02:49.197519 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.197409 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 02 19:02:49.197775 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.197763 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 19:02:49.197807 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.197785 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 19:02:49.201333 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.201319 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 02 19:02:49.203219 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.203207 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 02 19:02:49.203272 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.203236 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 02 19:02:49.204374 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.204346 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 02 19:02:49.204418 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.204386 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 02 19:02:49.204418 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.204395 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 02 19:02:49.204418 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.204403 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 02 19:02:49.204499 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.204446 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 02 19:02:49.204499 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.204452 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 02 19:02:49.204499 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.204460 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 02 19:02:49.204499 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.204465 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 02 19:02:49.204499 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.204480 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 02 19:02:49.204499 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.204488 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 02 19:02:49.204642 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.204501 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 02 19:02:49.204642 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.204507 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 02 19:02:49.205239 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.205229 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 02 19:02:49.205239 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.205240 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 02 19:02:49.207834 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.207784 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9sjnm" Jul 02 19:02:49.209386 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.209370 2565 server.go:1268] "Started kubelet" Jul 02 19:02:49.209481 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.209433 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 02 19:02:49.209656 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.209622 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 02 19:02:49.209716 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.209678 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 02 19:02:49.210461 ip-10-0-142-150 systemd[1]: Started Kubernetes Kubelet. Jul 02 19:02:49.210751 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.210730 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-150.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 02 19:02:49.211285 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.210955 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-150.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 02 19:02:49.211285 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.210723 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 02 19:02:49.211807 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.211647 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 02 19:02:49.213494 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.213151 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 02 19:02:49.213494 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.213206 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9sjnm" Jul 02 19:02:49.215101 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.215082 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 02 19:02:49.217300 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.216263 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-150.ec2.internal.18be8e4e3f80c8ae default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-150.ec2.internal,UID:ip-10-0-142-150.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-150.ec2.internal,},FirstTimestamp:2026-07-02 19:02:49.209292974 +0000 UTC m=+0.378654170,LastTimestamp:2026-07-02 19:02:49.209292974 +0000 UTC m=+0.378654170,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-150.ec2.internal,}" Jul 02 19:02:49.218627 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.218610 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 02 19:02:49.218627 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.218621 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 02 19:02:49.218774 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.218687 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:49.219257 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.219206 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 02 19:02:49.219257 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.219208 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 02 19:02:49.219257 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.219236 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 02 19:02:49.219442 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.219393 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 02 19:02:49.219442 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.219402 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 02 19:02:49.219442 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.219406 2565 factory.go:55] Registering systemd factory Jul 02 19:02:49.219442 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.219433 2565 factory.go:223] Registration of the systemd container factory successfully Jul 02 19:02:49.219825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.219704 2565 factory.go:153] Registering CRI-O factory Jul 02 19:02:49.219825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.219719 2565 factory.go:223] Registration of the crio container factory successfully Jul 02 19:02:49.219825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.219773 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 02 19:02:49.219825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.219800 2565 factory.go:103] Registering Raw factory Jul 02 19:02:49.219825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.219813 2565 manager.go:1196] Started watching for new ooms in manager Jul 02 19:02:49.221846 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.220393 2565 manager.go:349] Starting recovery of all containers Jul 02 19:02:49.231247 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.231067 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:02:49.231935 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.231883 2565 manager.go:354] Recovery completed Jul 02 19:02:49.234205 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.234185 2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-150.ec2.internal\" not found" node="ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.236309 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.236297 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:02:49.238065 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.238050 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:02:49.238142 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.238112 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:02:49.238142 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.238130 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:02:49.238631 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.238619 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 02 19:02:49.238693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.238633 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 02 19:02:49.238693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.238638 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:02:49.238693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.238653 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 02 19:02:49.239309 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.239294 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:02:49.239402 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.239320 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:02:49.239402 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.239330 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:02:49.240169 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.240128 2565 policy_none.go:49] "None policy: Start" Jul 02 19:02:49.240169 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.240144 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 02 19:02:49.240169 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.240154 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 02 19:02:49.241247 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.241231 2565 policy_none.go:47] "Start" Jul 02 19:02:49.276936 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.276917 2565 manager.go:337] "Starting Device Plugin manager" Jul 02 19:02:49.277021 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.276962 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 02 19:02:49.277021 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.276971 2565 server.go:87] "Starting device plugin registration server" Jul 02 19:02:49.277242 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.277230 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 02 19:02:49.277310 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.277244 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 02 19:02:49.277396 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.277377 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 02 19:02:49.277485 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.277473 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 02 19:02:49.277534 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.277485 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 02 19:02:49.277997 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.277980 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 02 19:02:49.278060 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.278028 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:49.377928 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.377882 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:02:49.379555 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.378948 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:02:49.379555 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.378981 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:02:49.379555 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.378996 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:02:49.379555 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.379025 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.383250 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.383221 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 02 19:02:49.384626 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.384606 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 02 19:02:49.384708 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.384644 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 02 19:02:49.384708 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.384664 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 02 19:02:49.384708 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.384697 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 02 19:02:49.385253 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.385229 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.385330 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.385254 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-150.ec2.internal\": node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:49.387086 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.387067 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:02:49.420407 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.420329 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:49.485464 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.485414 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-150.ec2.internal"] Jul 02 19:02:49.485534 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.485515 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:02:49.486492 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.486474 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:02:49.486545 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.486514 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:02:49.486545 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.486527 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:02:49.488108 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.488097 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:02:49.488229 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.488216 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.488265 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.488249 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:02:49.488871 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.488851 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:02:49.488960 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.488852 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:02:49.488960 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.488910 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:02:49.488960 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.488924 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:02:49.488960 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.488881 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:02:49.488960 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.488951 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:02:49.490493 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.490476 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.490565 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.490510 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:02:49.491155 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.491129 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:02:49.491263 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.491168 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:02:49.491263 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.491181 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:02:49.517164 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.517137 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-150.ec2.internal\" not found" node="ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.520076 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.520049 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/857dad294ddde2e2b97208e4a9c2f3af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal\" (UID: \"857dad294ddde2e2b97208e4a9c2f3af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.520145 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.520090 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/857dad294ddde2e2b97208e4a9c2f3af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal\" (UID: \"857dad294ddde2e2b97208e4a9c2f3af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.520145 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.520123 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4dee40681f34b7bc84fa53a0b9970f9c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-150.ec2.internal\" (UID: \"4dee40681f34b7bc84fa53a0b9970f9c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.520503 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.520488 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:49.521759 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.521743 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-150.ec2.internal\" not found" node="ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.620567 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.620538 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/857dad294ddde2e2b97208e4a9c2f3af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal\" (UID: \"857dad294ddde2e2b97208e4a9c2f3af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.620682 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.620586 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/857dad294ddde2e2b97208e4a9c2f3af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal\" (UID: \"857dad294ddde2e2b97208e4a9c2f3af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.620682 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.620604 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4dee40681f34b7bc84fa53a0b9970f9c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-150.ec2.internal\" (UID: \"4dee40681f34b7bc84fa53a0b9970f9c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.620682 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.620642 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/857dad294ddde2e2b97208e4a9c2f3af-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal\" (UID: \"857dad294ddde2e2b97208e4a9c2f3af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.620682 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.620651 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/857dad294ddde2e2b97208e4a9c2f3af-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal\" (UID: \"857dad294ddde2e2b97208e4a9c2f3af\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.620682 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:49.620668 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4dee40681f34b7bc84fa53a0b9970f9c-config\") pod \"kube-apiserver-proxy-ip-10-0-142-150.ec2.internal\" (UID: \"4dee40681f34b7bc84fa53a0b9970f9c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-150.ec2.internal" Jul 02 19:02:49.620836 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.620657 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:49.721289 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:49.721225 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:51.194129 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:51.194101 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:51.195194 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:51.195179 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal" Jul 02 19:02:51.197364 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:51.197348 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-150.ec2.internal" Jul 02 19:02:51.294696 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:51.294657 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:51.395123 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:51.395038 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:51.495450 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:51.495426 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:51.530631 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:51.530601 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 02 19:02:51.531157 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:51.530780 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 19:02:51.531157 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:51.530810 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 19:02:51.591226 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:51.591169 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 18:57:50 +0000 UTC" deadline="2028-04-09 04:13:28.777481168 +0000 UTC" Jul 02 19:02:51.591226 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:51.591218 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15513h10m37.186266739s" Jul 02 19:02:51.595770 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:51.595751 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:51.696247 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:51.696178 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:51.796499 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:51.796475 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:51.887647 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:51.887606 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857dad294ddde2e2b97208e4a9c2f3af.slice/crio-4a64cd4e2db9dc611f809333fa79fbc31f8483c76cac2b38b1231793fb201e30 WatchSource:0}: Error finding container 4a64cd4e2db9dc611f809333fa79fbc31f8483c76cac2b38b1231793fb201e30: Status 404 returned error can't find the container with id 4a64cd4e2db9dc611f809333fa79fbc31f8483c76cac2b38b1231793fb201e30 Jul 02 19:02:51.893349 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:51.893334 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:02:51.897062 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:51.897047 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:51.973611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:51.973526 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:02:51.997946 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:51.997914 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:52.098645 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:52.098612 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-150.ec2.internal\" not found" Jul 02 19:02:52.148891 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.147697 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:02:52.191925 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.191900 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal" Jul 02 19:02:52.206073 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.206045 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 19:02:52.207091 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.207078 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-150.ec2.internal" Jul 02 19:02:52.219917 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.219894 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 19:02:52.507800 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:52.507774 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dee40681f34b7bc84fa53a0b9970f9c.slice/crio-f08c74d6ab3dece84e0bf7364d4cf42f0f914a9591f047f397d7229c0d48368c WatchSource:0}: Error finding container f08c74d6ab3dece84e0bf7364d4cf42f0f914a9591f047f397d7229c0d48368c: Status 404 returned error can't find the container with id f08c74d6ab3dece84e0bf7364d4cf42f0f914a9591f047f397d7229c0d48368c Jul 02 19:02:52.571790 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.571752 2565 apiserver.go:52] "Watching apiserver" Jul 02 19:02:52.578080 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.578055 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 02 19:02:52.578457 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.578437 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-t7n9j","kube-system/konnectivity-agent-c485v","kube-system/kube-apiserver-proxy-ip-10-0-142-150.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq","openshift-cluster-node-tuning-operator/tuned-fp8qp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal","openshift-multus/network-metrics-daemon-c2q69","openshift-network-operator/iptables-alerter-2m8c5","openshift-dns/node-resolver-vd52d","openshift-image-registry/node-ca-rkjd9","openshift-multus/multus-additional-cni-plugins-kdm4z","openshift-multus/multus-zxctb","openshift-network-diagnostics/network-check-target-xmmlt","openshift-ovn-kubernetes/ovnkube-node-wx9nx"] Jul 02 19:02:52.580341 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.580319 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rkjd9" Jul 02 19:02:52.581960 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.581942 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.582879 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.582857 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 02 19:02:52.582977 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.582857 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8kgh\"" Jul 02 19:02:52.582977 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.582965 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 02 19:02:52.583108 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.583020 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 02 19:02:52.583321 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.583306 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2m8c5" Jul 02 19:02:52.584327 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.584310 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 02 19:02:52.584497 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.584475 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 02 19:02:52.584642 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.584628 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.584740 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.584726 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 02 19:02:52.584787 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.584750 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 02 19:02:52.585162 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.585023 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jrfbx\"" Jul 02 19:02:52.585162 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.585150 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 02 19:02:52.585549 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.585531 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:02:52.585705 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.585674 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zbqwz\"" Jul 02 19:02:52.585762 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.585742 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 02 19:02:52.585981 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.585964 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:02:52.586055 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:52.586031 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:02:52.586112 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.585969 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 02 19:02:52.587005 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.586985 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 02 19:02:52.587283 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.587267 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c485v" Jul 02 19:02:52.587568 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.587548 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-m5v75\"" Jul 02 19:02:52.588478 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.588461 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vd52d" Jul 02 19:02:52.589831 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.589466 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7kzlv\"" Jul 02 19:02:52.589831 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.589472 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 02 19:02:52.589831 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.589586 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 02 19:02:52.589831 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.589794 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:02:52.590186 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:52.589887 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:02:52.591277 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.591260 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 02 19:02:52.592192 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.592173 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-srlf6\"" Jul 02 19:02:52.592271 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.592250 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 02 19:02:52.592271 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.592266 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 02 19:02:52.593023 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.592661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:02:52.593023 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:52.592739 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:02:52.594744 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.594670 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.597031 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.596514 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.597205 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.597189 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 02 19:02:52.597608 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.597577 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:02:52.597693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.597653 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jhwbd\"" Jul 02 19:02:52.597874 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.597833 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.598582 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.598562 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 02 19:02:52.598776 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.598759 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8xlqd\"" Jul 02 19:02:52.598981 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.598965 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 02 19:02:52.599179 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.599163 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 02 19:02:52.600115 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600093 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-kubernetes\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.600331 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-host\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.600331 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600152 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-cnibin\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.600331 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600169 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 02 19:02:52.600331 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600176 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-var-lib-cni-bin\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.600331 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600200 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc4tw\" (UniqueName: \"kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw\") pod \"network-check-target-xmmlt\" (UID: \"0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48\") " pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:02:52.600331 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600224 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-os-release\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.600331 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600245 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-var-lib-kubelet\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.600331 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600267 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-sysconfig\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.600331 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600292 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbdrw\" (UniqueName: \"kubernetes.io/projected/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-kube-api-access-qbdrw\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.600331 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600314 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-system-cni-dir\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.600747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600347 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-run-k8s-cni-cncf-io\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.600747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600378 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjfnz\" (UniqueName: \"kubernetes.io/projected/f583ae21-02a0-4d33-b57c-b8370bbaecdc-kube-api-access-vjfnz\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.600747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600399 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35b8716c-c781-4444-85ea-b906b33f52fb-host\") pod \"node-ca-rkjd9\" (UID: \"35b8716c-c781-4444-85ea-b906b33f52fb\") " pod="openshift-image-registry/node-ca-rkjd9" Jul 02 19:02:52.600747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600423 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2x8b\" (UniqueName: \"kubernetes.io/projected/35b8716c-c781-4444-85ea-b906b33f52fb-kube-api-access-c2x8b\") pod \"node-ca-rkjd9\" (UID: \"35b8716c-c781-4444-85ea-b906b33f52fb\") " pod="openshift-image-registry/node-ca-rkjd9" Jul 02 19:02:52.600747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600447 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.600747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600481 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.600747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600506 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f01d507-0625-477b-bf2f-7382a4c5bc9b-kubelet-config\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:02:52.600747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600529 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-var-lib-kubelet\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.600747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600558 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-multus-socket-dir-parent\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.600747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600584 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr2s8\" (UniqueName: \"kubernetes.io/projected/cf954a6b-7e4e-472a-8755-eafa3367d32c-kube-api-access-xr2s8\") pod \"network-metrics-daemon-c2q69\" (UID: \"cf954a6b-7e4e-472a-8755-eafa3367d32c\") " pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:02:52.600747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600607 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 02 19:02:52.600747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600402 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 02 19:02:52.600747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600609 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05ad455a-193e-433c-994e-44600efc8041-agent-certs\") pod \"konnectivity-agent-c485v\" (UID: \"05ad455a-193e-433c-994e-44600efc8041\") " pod="kube-system/konnectivity-agent-c485v" Jul 02 19:02:52.601216 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600785 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05ad455a-193e-433c-994e-44600efc8041-konnectivity-ca\") pod \"konnectivity-agent-c485v\" (UID: \"05ad455a-193e-433c-994e-44600efc8041\") " pod="kube-system/konnectivity-agent-c485v" Jul 02 19:02:52.601216 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600810 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/562e2387-8a49-4d49-b44a-4006eb653991-etc-tuned\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.601216 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600836 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-sysctl-d\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.601216 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstl5\" (UniqueName: \"kubernetes.io/projected/bb5fb82b-d3ee-4078-bd9b-391072dbb2f6-kube-api-access-xstl5\") pod \"iptables-alerter-2m8c5\" (UID: \"bb5fb82b-d3ee-4078-bd9b-391072dbb2f6\") " pod="openshift-network-operator/iptables-alerter-2m8c5" Jul 02 19:02:52.601216 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600935 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-system-cni-dir\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.601216 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.600998 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb5fb82b-d3ee-4078-bd9b-391072dbb2f6-host-slash\") pod \"iptables-alerter-2m8c5\" (UID: \"bb5fb82b-d3ee-4078-bd9b-391072dbb2f6\") " pod="openshift-network-operator/iptables-alerter-2m8c5" Jul 02 19:02:52.601216 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601022 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-hostroot\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.601216 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601046 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-sys\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.601216 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601068 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-multus-cni-dir\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.601216 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601116 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 02 19:02:52.601535 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601274 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 02 19:02:52.601535 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601118 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35b8716c-c781-4444-85ea-b906b33f52fb-serviceca\") pod \"node-ca-rkjd9\" (UID: \"35b8716c-c781-4444-85ea-b906b33f52fb\") " pod="openshift-image-registry/node-ca-rkjd9" Jul 02 19:02:52.601535 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601389 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.601535 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601414 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a35ed1-ba9b-41d9-bf9b-bd99874366c6-tmp-dir\") pod \"node-resolver-vd52d\" (UID: \"57a35ed1-ba9b-41d9-bf9b-bd99874366c6\") " pod="openshift-dns/node-resolver-vd52d" Jul 02 19:02:52.601535 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601439 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-run\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.601535 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601499 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-sysctl-conf\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.601535 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601523 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/562e2387-8a49-4d49-b44a-4006eb653991-tmp\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.601791 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601537 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 02 19:02:52.601791 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601543 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-systemd\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.601791 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601565 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb5fb82b-d3ee-4078-bd9b-391072dbb2f6-iptables-alerter-script\") pod \"iptables-alerter-2m8c5\" (UID: \"bb5fb82b-d3ee-4078-bd9b-391072dbb2f6\") " pod="openshift-network-operator/iptables-alerter-2m8c5" Jul 02 19:02:52.601791 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601587 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57a35ed1-ba9b-41d9-bf9b-bd99874366c6-hosts-file\") pod \"node-resolver-vd52d\" (UID: \"57a35ed1-ba9b-41d9-bf9b-bd99874366c6\") " pod="openshift-dns/node-resolver-vd52d" Jul 02 19:02:52.601791 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601614 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:02:52.601791 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601638 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-etc-kubernetes\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.601791 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601658 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs\") pod \"network-metrics-daemon-c2q69\" (UID: \"cf954a6b-7e4e-472a-8755-eafa3367d32c\") " pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:02:52.601791 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601658 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-88g9k\"" Jul 02 19:02:52.601791 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601684 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w92ht\" (UniqueName: \"kubernetes.io/projected/57a35ed1-ba9b-41d9-bf9b-bd99874366c6-kube-api-access-w92ht\") pod \"node-resolver-vd52d\" (UID: \"57a35ed1-ba9b-41d9-bf9b-bd99874366c6\") " pod="openshift-dns/node-resolver-vd52d" Jul 02 19:02:52.601791 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601759 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-modprobe-d\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.602183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601788 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f01d507-0625-477b-bf2f-7382a4c5bc9b-dbus\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:02:52.602183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601826 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f583ae21-02a0-4d33-b57c-b8370bbaecdc-cni-binary-copy\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.602183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601862 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-run-multus-certs\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.602183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601880 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-lib-modules\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.602183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601896 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-cnibin\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.602183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601912 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.602183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601935 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-run-netns\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.602183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601954 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-os-release\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.602183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601975 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-multus-conf-dir\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.602183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.601997 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f583ae21-02a0-4d33-b57c-b8370bbaecdc-multus-daemon-config\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.602183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.602023 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chzwm\" (UniqueName: \"kubernetes.io/projected/562e2387-8a49-4d49-b44a-4006eb653991-kube-api-access-chzwm\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.602183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.602039 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-var-lib-cni-multus\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.605044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.605025 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 19:02:52.623269 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.623243 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x6gzn" Jul 02 19:02:52.638799 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.638775 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x6gzn" Jul 02 19:02:52.686260 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.686234 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:02:52.693018 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.692993 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 02 19:02:52.702932 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.702907 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-var-lib-openvswitch\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.702932 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.702937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-ovn-node-metrics-cert\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.703161 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.702960 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-sysctl-d\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.703161 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.702997 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstl5\" (UniqueName: \"kubernetes.io/projected/bb5fb82b-d3ee-4078-bd9b-391072dbb2f6-kube-api-access-xstl5\") pod \"iptables-alerter-2m8c5\" (UID: \"bb5fb82b-d3ee-4078-bd9b-391072dbb2f6\") " pod="openshift-network-operator/iptables-alerter-2m8c5" Jul 02 19:02:52.703161 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703023 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-system-cni-dir\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.703161 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703079 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-system-cni-dir\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.703161 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703096 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb5fb82b-d3ee-4078-bd9b-391072dbb2f6-host-slash\") pod \"iptables-alerter-2m8c5\" (UID: \"bb5fb82b-d3ee-4078-bd9b-391072dbb2f6\") " pod="openshift-network-operator/iptables-alerter-2m8c5" Jul 02 19:02:52.703161 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703141 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-hostroot\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.703452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703177 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb5fb82b-d3ee-4078-bd9b-391072dbb2f6-host-slash\") pod \"iptables-alerter-2m8c5\" (UID: \"bb5fb82b-d3ee-4078-bd9b-391072dbb2f6\") " pod="openshift-network-operator/iptables-alerter-2m8c5" Jul 02 19:02:52.703452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703183 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-run-netns\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.703452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703210 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-etc-openvswitch\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.703452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703219 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-sysctl-d\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.703452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703246 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-hostroot\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.703452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703269 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-sys\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.703452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703305 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-systemd-units\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.703452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703331 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-node-log\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.703452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703358 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-sys\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.703452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703363 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-log-socket\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.703452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703408 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-run-ovn-kubernetes\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.703452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703440 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.704044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-ovnkube-script-lib\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.704044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703510 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-multus-cni-dir\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.704044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703565 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-multus-cni-dir\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.704044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703596 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35b8716c-c781-4444-85ea-b906b33f52fb-serviceca\") pod \"node-ca-rkjd9\" (UID: \"35b8716c-c781-4444-85ea-b906b33f52fb\") " pod="openshift-image-registry/node-ca-rkjd9" Jul 02 19:02:52.704044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703649 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.704044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703743 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a35ed1-ba9b-41d9-bf9b-bd99874366c6-tmp-dir\") pod \"node-resolver-vd52d\" (UID: \"57a35ed1-ba9b-41d9-bf9b-bd99874366c6\") " pod="openshift-dns/node-resolver-vd52d" Jul 02 19:02:52.704044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703888 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-run\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.704044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703922 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-device-dir\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.704044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703949 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-sysctl-conf\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.704044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703977 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/562e2387-8a49-4d49-b44a-4006eb653991-tmp\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.704044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.703982 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-run\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.704044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704000 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704025 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-registration-dir\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704080 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704096 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a35ed1-ba9b-41d9-bf9b-bd99874366c6-tmp-dir\") pod \"node-resolver-vd52d\" (UID: \"57a35ed1-ba9b-41d9-bf9b-bd99874366c6\") " pod="openshift-dns/node-resolver-vd52d" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704103 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-systemd\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704132 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-sysctl-conf\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704149 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/35b8716c-c781-4444-85ea-b906b33f52fb-serviceca\") pod \"node-ca-rkjd9\" (UID: \"35b8716c-c781-4444-85ea-b906b33f52fb\") " pod="openshift-image-registry/node-ca-rkjd9" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704168 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb5fb82b-d3ee-4078-bd9b-391072dbb2f6-iptables-alerter-script\") pod \"iptables-alerter-2m8c5\" (UID: \"bb5fb82b-d3ee-4078-bd9b-391072dbb2f6\") " pod="openshift-network-operator/iptables-alerter-2m8c5" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704209 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57a35ed1-ba9b-41d9-bf9b-bd99874366c6-hosts-file\") pod \"node-resolver-vd52d\" (UID: \"57a35ed1-ba9b-41d9-bf9b-bd99874366c6\") " pod="openshift-dns/node-resolver-vd52d" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704235 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-systemd\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704312 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704333 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57a35ed1-ba9b-41d9-bf9b-bd99874366c6-hosts-file\") pod \"node-resolver-vd52d\" (UID: \"57a35ed1-ba9b-41d9-bf9b-bd99874366c6\") " pod="openshift-dns/node-resolver-vd52d" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704372 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-etc-kubernetes\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704398 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs\") pod \"network-metrics-daemon-c2q69\" (UID: \"cf954a6b-7e4e-472a-8755-eafa3367d32c\") " pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704459 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704468 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704468 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-etc-kubernetes\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:52.704499 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704528 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w92ht\" (UniqueName: \"kubernetes.io/projected/57a35ed1-ba9b-41d9-bf9b-bd99874366c6-kube-api-access-w92ht\") pod \"node-resolver-vd52d\" (UID: \"57a35ed1-ba9b-41d9-bf9b-bd99874366c6\") " pod="openshift-dns/node-resolver-vd52d" Jul 02 19:02:52.704585 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:52.704536 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:02:52.705428 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:52.704642 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs podName:cf954a6b-7e4e-472a-8755-eafa3367d32c nodeName:}" failed. No retries permitted until 2026-07-02 19:02:53.204621913 +0000 UTC m=+3.001492083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs") pod "network-metrics-daemon-c2q69" (UID: "cf954a6b-7e4e-472a-8755-eafa3367d32c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:02:52.705428 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:52.704663 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret podName:6f01d507-0625-477b-bf2f-7382a4c5bc9b nodeName:}" failed. No retries permitted until 2026-07-02 19:02:53.20465217 +0000 UTC m=+3.001522353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret") pod "global-pull-secret-syncer-t7n9j" (UID: "6f01d507-0625-477b-bf2f-7382a4c5bc9b") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:02:52.705428 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-modprobe-d\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.705428 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704721 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bb5fb82b-d3ee-4078-bd9b-391072dbb2f6-iptables-alerter-script\") pod \"iptables-alerter-2m8c5\" (UID: \"bb5fb82b-d3ee-4078-bd9b-391072dbb2f6\") " pod="openshift-network-operator/iptables-alerter-2m8c5" Jul 02 19:02:52.705428 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704792 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-run-openvswitch\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.705428 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f01d507-0625-477b-bf2f-7382a4c5bc9b-dbus\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:02:52.705428 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704856 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f583ae21-02a0-4d33-b57c-b8370bbaecdc-cni-binary-copy\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.705428 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704870 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-modprobe-d\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.705428 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704876 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-sys-fs\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.705428 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704895 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f01d507-0625-477b-bf2f-7382a4c5bc9b-dbus\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:02:52.705428 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704916 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-run-multus-certs\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.705428 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.704944 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-lib-modules\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.706173 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.706066 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f583ae21-02a0-4d33-b57c-b8370bbaecdc-cni-binary-copy\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.707355 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.707320 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-run-multus-certs\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.707573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.707556 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/562e2387-8a49-4d49-b44a-4006eb653991-tmp\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.707697 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.707678 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-lib-modules\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.707858 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.707755 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-cnibin\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.707858 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.707834 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-cnibin\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.710313 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.710275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.710428 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.710347 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-run-netns\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.710488 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.710467 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-os-release\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.710560 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.710479 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-os-release\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.710696 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.710641 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-multus-conf-dir\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.710767 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.710658 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-run-netns\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.710767 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.710700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f583ae21-02a0-4d33-b57c-b8370bbaecdc-multus-daemon-config\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.710767 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.710758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chzwm\" (UniqueName: \"kubernetes.io/projected/562e2387-8a49-4d49-b44a-4006eb653991-kube-api-access-chzwm\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.710936 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.710767 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-multus-conf-dir\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.710936 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.710798 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-socket-dir\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.710936 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.710865 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-kubelet\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.710936 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.710906 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-run-ovn\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.711129 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.710937 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-cni-binary-copy\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.711129 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.710951 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-env-overrides\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.711129 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.710986 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-var-lib-cni-multus\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.711129 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711030 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-kubernetes\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.711129 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711079 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-var-lib-cni-multus\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.711129 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711087 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-host\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.711385 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-cnibin\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.711385 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711168 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-var-lib-cni-bin\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.711385 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711176 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-kubernetes\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.711385 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711200 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-host\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.711385 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711226 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4tw\" (UniqueName: \"kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw\") pod \"network-check-target-xmmlt\" (UID: \"0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48\") " pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:02:52.711385 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711259 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-os-release\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.711385 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711268 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-var-lib-cni-bin\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.711385 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711280 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-cnibin\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.711385 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711293 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-var-lib-kubelet\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.711385 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711330 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-slash\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.711855 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711379 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f583ae21-02a0-4d33-b57c-b8370bbaecdc-multus-daemon-config\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.711855 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn6gr\" (UniqueName: \"kubernetes.io/projected/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-kube-api-access-nn6gr\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.711855 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711438 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-var-lib-kubelet\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.711855 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711455 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-os-release\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.711855 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711538 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-sysconfig\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.711855 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711586 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbdrw\" (UniqueName: \"kubernetes.io/projected/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-kube-api-access-qbdrw\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.711855 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.711667 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/562e2387-8a49-4d49-b44a-4006eb653991-etc-sysconfig\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.712215 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712198 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-system-cni-dir\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.712296 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712235 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-run-k8s-cni-cncf-io\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.712296 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712263 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjfnz\" (UniqueName: \"kubernetes.io/projected/f583ae21-02a0-4d33-b57c-b8370bbaecdc-kube-api-access-vjfnz\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.712296 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712286 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35b8716c-c781-4444-85ea-b906b33f52fb-host\") pod \"node-ca-rkjd9\" (UID: \"35b8716c-c781-4444-85ea-b906b33f52fb\") " pod="openshift-image-registry/node-ca-rkjd9" Jul 02 19:02:52.712456 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712311 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2x8b\" (UniqueName: \"kubernetes.io/projected/35b8716c-c781-4444-85ea-b906b33f52fb-kube-api-access-c2x8b\") pod \"node-ca-rkjd9\" (UID: \"35b8716c-c781-4444-85ea-b906b33f52fb\") " pod="openshift-image-registry/node-ca-rkjd9" Jul 02 19:02:52.712456 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712339 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5khf\" (UniqueName: \"kubernetes.io/projected/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-kube-api-access-q5khf\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.712456 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712357 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-run-k8s-cni-cncf-io\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.712456 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712366 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.712456 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712395 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.712456 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712414 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35b8716c-c781-4444-85ea-b906b33f52fb-host\") pod \"node-ca-rkjd9\" (UID: \"35b8716c-c781-4444-85ea-b906b33f52fb\") " pod="openshift-image-registry/node-ca-rkjd9" Jul 02 19:02:52.712456 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712415 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-system-cni-dir\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.712456 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712423 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f01d507-0625-477b-bf2f-7382a4c5bc9b-kubelet-config\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:02:52.712825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-var-lib-kubelet\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.712825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712491 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-run-systemd\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.712825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712513 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-cni-bin\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.712825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712534 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-cni-netd\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.712825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712560 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-ovnkube-config\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.712825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712584 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-multus-socket-dir-parent\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.712825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr2s8\" (UniqueName: \"kubernetes.io/projected/cf954a6b-7e4e-472a-8755-eafa3367d32c-kube-api-access-xr2s8\") pod \"network-metrics-daemon-c2q69\" (UID: \"cf954a6b-7e4e-472a-8755-eafa3367d32c\") " pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:02:52.712825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712634 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05ad455a-193e-433c-994e-44600efc8041-agent-certs\") pod \"konnectivity-agent-c485v\" (UID: \"05ad455a-193e-433c-994e-44600efc8041\") " pod="kube-system/konnectivity-agent-c485v" Jul 02 19:02:52.712825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712657 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05ad455a-193e-433c-994e-44600efc8041-konnectivity-ca\") pod \"konnectivity-agent-c485v\" (UID: \"05ad455a-193e-433c-994e-44600efc8041\") " pod="kube-system/konnectivity-agent-c485v" Jul 02 19:02:52.712825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/562e2387-8a49-4d49-b44a-4006eb653991-etc-tuned\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.712825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712715 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f01d507-0625-477b-bf2f-7382a4c5bc9b-kubelet-config\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:02:52.712825 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712788 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-multus-socket-dir-parent\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.713379 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.712958 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.713379 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.713001 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f583ae21-02a0-4d33-b57c-b8370bbaecdc-host-var-lib-kubelet\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.713379 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.713061 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstl5\" (UniqueName: \"kubernetes.io/projected/bb5fb82b-d3ee-4078-bd9b-391072dbb2f6-kube-api-access-xstl5\") pod \"iptables-alerter-2m8c5\" (UID: \"bb5fb82b-d3ee-4078-bd9b-391072dbb2f6\") " pod="openshift-network-operator/iptables-alerter-2m8c5" Jul 02 19:02:52.713379 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.713231 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.713379 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.713313 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05ad455a-193e-433c-994e-44600efc8041-konnectivity-ca\") pod \"konnectivity-agent-c485v\" (UID: \"05ad455a-193e-433c-994e-44600efc8041\") " pod="kube-system/konnectivity-agent-c485v" Jul 02 19:02:52.715492 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.715391 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05ad455a-193e-433c-994e-44600efc8041-agent-certs\") pod \"konnectivity-agent-c485v\" (UID: \"05ad455a-193e-433c-994e-44600efc8041\") " pod="kube-system/konnectivity-agent-c485v" Jul 02 19:02:52.715492 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.715461 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/562e2387-8a49-4d49-b44a-4006eb653991-etc-tuned\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.716707 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.716684 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w92ht\" (UniqueName: \"kubernetes.io/projected/57a35ed1-ba9b-41d9-bf9b-bd99874366c6-kube-api-access-w92ht\") pod \"node-resolver-vd52d\" (UID: \"57a35ed1-ba9b-41d9-bf9b-bd99874366c6\") " pod="openshift-dns/node-resolver-vd52d" Jul 02 19:02:52.718413 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:52.718398 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:02:52.718413 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:52.718415 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:02:52.718548 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:52.718425 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hc4tw for pod openshift-network-diagnostics/network-check-target-xmmlt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:02:52.718548 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:52.718473 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw podName:0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48 nodeName:}" failed. No retries permitted until 2026-07-02 19:02:53.218458275 +0000 UTC m=+3.015328448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hc4tw" (UniqueName: "kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw") pod "network-check-target-xmmlt" (UID: "0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:02:52.721037 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.721014 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chzwm\" (UniqueName: \"kubernetes.io/projected/562e2387-8a49-4d49-b44a-4006eb653991-kube-api-access-chzwm\") pod \"tuned-fp8qp\" (UID: \"562e2387-8a49-4d49-b44a-4006eb653991\") " pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.722283 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.722266 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr2s8\" (UniqueName: \"kubernetes.io/projected/cf954a6b-7e4e-472a-8755-eafa3367d32c-kube-api-access-xr2s8\") pod \"network-metrics-daemon-c2q69\" (UID: \"cf954a6b-7e4e-472a-8755-eafa3367d32c\") " pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:02:52.722409 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.722389 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbdrw\" (UniqueName: \"kubernetes.io/projected/7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab-kube-api-access-qbdrw\") pod \"multus-additional-cni-plugins-kdm4z\" (UID: \"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab\") " pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.722711 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.722685 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2x8b\" (UniqueName: \"kubernetes.io/projected/35b8716c-c781-4444-85ea-b906b33f52fb-kube-api-access-c2x8b\") pod \"node-ca-rkjd9\" (UID: \"35b8716c-c781-4444-85ea-b906b33f52fb\") " pod="openshift-image-registry/node-ca-rkjd9" Jul 02 19:02:52.722981 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.722962 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjfnz\" (UniqueName: \"kubernetes.io/projected/f583ae21-02a0-4d33-b57c-b8370bbaecdc-kube-api-access-vjfnz\") pod \"multus-zxctb\" (UID: \"f583ae21-02a0-4d33-b57c-b8370bbaecdc\") " pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.761332 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.761240 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-150.ec2.internal" event={"ID":"4dee40681f34b7bc84fa53a0b9970f9c","Type":"ContainerStarted","Data":"f08c74d6ab3dece84e0bf7364d4cf42f0f914a9591f047f397d7229c0d48368c"} Jul 02 19:02:52.762296 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.762274 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal" event={"ID":"857dad294ddde2e2b97208e4a9c2f3af","Type":"ContainerStarted","Data":"4a64cd4e2db9dc611f809333fa79fbc31f8483c76cac2b38b1231793fb201e30"} Jul 02 19:02:52.813573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5khf\" (UniqueName: \"kubernetes.io/projected/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-kube-api-access-q5khf\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.813573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-run-systemd\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813423 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-cni-bin\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813437 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-cni-netd\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813452 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-ovnkube-config\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813470 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-var-lib-openvswitch\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813485 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-ovn-node-metrics-cert\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813503 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-run-netns\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813518 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-etc-openvswitch\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813542 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-systemd-units\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813562 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-node-log\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813576 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-log-socket\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-run-ovn-kubernetes\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-ovnkube-script-lib\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813642 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-device-dir\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.813988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813657 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.813988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-registration-dir\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.813988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.813988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813724 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-run-openvswitch\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813740 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-sys-fs\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.813988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813760 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-socket-dir\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.813988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813775 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-kubelet\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813789 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-run-ovn\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-env-overrides\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-slash\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.813988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.813873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn6gr\" (UniqueName: \"kubernetes.io/projected/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-kube-api-access-nn6gr\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.814501 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.814213 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-run-systemd\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.814501 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.814246 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-cni-bin\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.814501 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.814269 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-cni-netd\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.814678 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.814660 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-ovnkube-config\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.814730 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.814711 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-var-lib-openvswitch\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.815028 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.815004 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.815136 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.815050 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-run-netns\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.815136 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.815078 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-etc-openvswitch\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.815136 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.815104 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-systemd-units\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.815136 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.815129 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-node-log\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.815326 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.815152 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-log-socket\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.815326 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.815177 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-run-ovn-kubernetes\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.815326 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.815203 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.815563 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.815548 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-ovnkube-script-lib\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.815619 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.815603 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-device-dir\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.815698 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.815687 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-socket-dir\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.815919 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.815904 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-registration-dir\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.816111 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.815979 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.816111 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.816006 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-run-openvswitch\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.816111 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.816051 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-sys-fs\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.816111 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.816091 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-run-ovn\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.816407 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.816126 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-kubelet\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.816633 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.816610 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-host-slash\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.816752 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.816688 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-env-overrides\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.817367 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.817346 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-ovn-node-metrics-cert\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.822725 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.822706 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn6gr\" (UniqueName: \"kubernetes.io/projected/a3fd3a8f-67dd-41f4-b313-1c855db78fc3-kube-api-access-nn6gr\") pod \"ovnkube-node-wx9nx\" (UID: \"a3fd3a8f-67dd-41f4-b313-1c855db78fc3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:52.824742 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.824718 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5khf\" (UniqueName: \"kubernetes.io/projected/73035ff3-c9bf-4f9e-9d60-40d7282f97c4-kube-api-access-q5khf\") pod \"aws-ebs-csi-driver-node-8ctnq\" (UID: \"73035ff3-c9bf-4f9e-9d60-40d7282f97c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.892579 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.892539 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rkjd9" Jul 02 19:02:52.900836 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.900810 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kdm4z" Jul 02 19:02:52.909780 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.909754 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2m8c5" Jul 02 19:02:52.916431 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.916406 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zxctb" Jul 02 19:02:52.923074 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.923053 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c485v" Jul 02 19:02:52.930855 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.930821 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vd52d" Jul 02 19:02:52.938416 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.938395 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" Jul 02 19:02:52.944113 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.944096 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" Jul 02 19:02:52.947752 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.947735 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:02:52.948890 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:52.948871 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:02:53.190339 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:53.190298 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73035ff3_c9bf_4f9e_9d60_40d7282f97c4.slice/crio-9d84cd68100802f75af746d845aca5a03eff89a130b64fb60646995c907b3815 WatchSource:0}: Error finding container 9d84cd68100802f75af746d845aca5a03eff89a130b64fb60646995c907b3815: Status 404 returned error can't find the container with id 9d84cd68100802f75af746d845aca5a03eff89a130b64fb60646995c907b3815 Jul 02 19:02:53.200451 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:53.200424 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562e2387_8a49_4d49_b44a_4006eb653991.slice/crio-bdd4eb4547d713b1df547bdc3f52825235ac332059343d493998bb0258f20d61 WatchSource:0}: Error finding container bdd4eb4547d713b1df547bdc3f52825235ac332059343d493998bb0258f20d61: Status 404 returned error can't find the container with id bdd4eb4547d713b1df547bdc3f52825235ac332059343d493998bb0258f20d61 Jul 02 19:02:53.201012 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:53.200983 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b8716c_c781_4444_85ea_b906b33f52fb.slice/crio-3778f0663127fe717d470e1f149f9d8d31594d3667c550f9c2dba68bc2acf9cc WatchSource:0}: Error finding container 3778f0663127fe717d470e1f149f9d8d31594d3667c550f9c2dba68bc2acf9cc: Status 404 returned error can't find the container with id 3778f0663127fe717d470e1f149f9d8d31594d3667c550f9c2dba68bc2acf9cc Jul 02 19:02:53.202051 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:53.202032 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c19a6c2_f663_4fd4_a5f9_5c39a74c26ab.slice/crio-cd440a99e1ddb60e3bcaaa6967989c2b5b46572f32a96d368cac835c6f49199f WatchSource:0}: Error finding container cd440a99e1ddb60e3bcaaa6967989c2b5b46572f32a96d368cac835c6f49199f: Status 404 returned error can't find the container with id cd440a99e1ddb60e3bcaaa6967989c2b5b46572f32a96d368cac835c6f49199f Jul 02 19:02:53.203556 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:53.203534 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a35ed1_ba9b_41d9_bf9b_bd99874366c6.slice/crio-89d631f922ff7bfb14455acbc5f926ebbfa17d88b22ab4130a264acf0c5ee233 WatchSource:0}: Error finding container 89d631f922ff7bfb14455acbc5f926ebbfa17d88b22ab4130a264acf0c5ee233: Status 404 returned error can't find the container with id 89d631f922ff7bfb14455acbc5f926ebbfa17d88b22ab4130a264acf0c5ee233 Jul 02 19:02:53.204283 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:53.204262 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5fb82b_d3ee_4078_bd9b_391072dbb2f6.slice/crio-5bd8018668f8900e4695a9e1b25f19ece4779752b187e2d46a451b60ef12793d WatchSource:0}: Error finding container 5bd8018668f8900e4695a9e1b25f19ece4779752b187e2d46a451b60ef12793d: Status 404 returned error can't find the container with id 5bd8018668f8900e4695a9e1b25f19ece4779752b187e2d46a451b60ef12793d Jul 02 19:02:53.205312 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:53.205215 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fd3a8f_67dd_41f4_b313_1c855db78fc3.slice/crio-216e1c13237956f049e5c3aaacc947eb15b42377e02840e96b158e5c93d062f9 WatchSource:0}: Error finding container 216e1c13237956f049e5c3aaacc947eb15b42377e02840e96b158e5c93d062f9: Status 404 returned error can't find the container with id 216e1c13237956f049e5c3aaacc947eb15b42377e02840e96b158e5c93d062f9 Jul 02 19:02:53.206516 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:53.206459 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf583ae21_02a0_4d33_b57c_b8370bbaecdc.slice/crio-d612d84e03deb0f89f764ed2a06f9ef85dbdd7220851150c9859ca64757b97aa WatchSource:0}: Error finding container d612d84e03deb0f89f764ed2a06f9ef85dbdd7220851150c9859ca64757b97aa: Status 404 returned error can't find the container with id d612d84e03deb0f89f764ed2a06f9ef85dbdd7220851150c9859ca64757b97aa Jul 02 19:02:53.207238 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:02:53.207202 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ad455a_193e_433c_994e_44600efc8041.slice/crio-fad51ebdfafdb9f5bb591885d33d1d6d3e0a4bebc4528cee8c650c99ceef41c8 WatchSource:0}: Error finding container fad51ebdfafdb9f5bb591885d33d1d6d3e0a4bebc4528cee8c650c99ceef41c8: Status 404 returned error can't find the container with id fad51ebdfafdb9f5bb591885d33d1d6d3e0a4bebc4528cee8c650c99ceef41c8 Jul 02 19:02:53.216348 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.216326 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:02:53.216431 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.216357 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs\") pod \"network-metrics-daemon-c2q69\" (UID: \"cf954a6b-7e4e-472a-8755-eafa3367d32c\") " pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:02:53.216508 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:53.216476 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:02:53.216542 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:53.216532 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret podName:6f01d507-0625-477b-bf2f-7382a4c5bc9b nodeName:}" failed. No retries permitted until 2026-07-02 19:02:54.216512909 +0000 UTC m=+4.013383099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret") pod "global-pull-secret-syncer-t7n9j" (UID: "6f01d507-0625-477b-bf2f-7382a4c5bc9b") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:02:53.216606 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:53.216591 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:02:53.216654 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:53.216642 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs podName:cf954a6b-7e4e-472a-8755-eafa3367d32c nodeName:}" failed. No retries permitted until 2026-07-02 19:02:54.216626751 +0000 UTC m=+4.013496919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs") pod "network-metrics-daemon-c2q69" (UID: "cf954a6b-7e4e-472a-8755-eafa3367d32c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:02:53.317265 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.317232 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4tw\" (UniqueName: \"kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw\") pod \"network-check-target-xmmlt\" (UID: \"0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48\") " pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:02:53.317431 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:53.317407 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:02:53.317431 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:53.317429 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:02:53.317539 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:53.317443 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hc4tw for pod openshift-network-diagnostics/network-check-target-xmmlt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:02:53.317539 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:53.317509 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw podName:0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48 nodeName:}" failed. No retries permitted until 2026-07-02 19:02:54.317489061 +0000 UTC m=+4.114359231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hc4tw" (UniqueName: "kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw") pod "network-check-target-xmmlt" (UID: "0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:02:53.641168 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.640918 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 18:57:52 +0000 UTC" deadline="2027-12-06 13:11:01.402807642 +0000 UTC" Jul 02 19:02:53.641168 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.641166 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12522h8m7.761645703s" Jul 02 19:02:53.769068 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.768953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" event={"ID":"73035ff3-c9bf-4f9e-9d60-40d7282f97c4","Type":"ContainerStarted","Data":"9d84cd68100802f75af746d845aca5a03eff89a130b64fb60646995c907b3815"} Jul 02 19:02:53.772107 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.772027 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zxctb" event={"ID":"f583ae21-02a0-4d33-b57c-b8370bbaecdc","Type":"ContainerStarted","Data":"d612d84e03deb0f89f764ed2a06f9ef85dbdd7220851150c9859ca64757b97aa"} Jul 02 19:02:53.777617 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.777534 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdm4z" event={"ID":"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab","Type":"ContainerStarted","Data":"cd440a99e1ddb60e3bcaaa6967989c2b5b46572f32a96d368cac835c6f49199f"} Jul 02 19:02:53.780252 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.780190 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rkjd9" event={"ID":"35b8716c-c781-4444-85ea-b906b33f52fb","Type":"ContainerStarted","Data":"3778f0663127fe717d470e1f149f9d8d31594d3667c550f9c2dba68bc2acf9cc"} Jul 02 19:02:53.784488 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.784456 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" event={"ID":"562e2387-8a49-4d49-b44a-4006eb653991","Type":"ContainerStarted","Data":"bdd4eb4547d713b1df547bdc3f52825235ac332059343d493998bb0258f20d61"} Jul 02 19:02:53.792790 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.792162 2565 generic.go:356] "Generic (PLEG): container finished" podID="857dad294ddde2e2b97208e4a9c2f3af" containerID="eaea11134faa279c9b777e18ef71defc4c6493a5f44a551a65e1f010b02708b9" exitCode=0 Jul 02 19:02:53.792790 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.792411 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal" event={"ID":"857dad294ddde2e2b97208e4a9c2f3af","Type":"ContainerDied","Data":"eaea11134faa279c9b777e18ef71defc4c6493a5f44a551a65e1f010b02708b9"} Jul 02 19:02:53.798085 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.797987 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c485v" event={"ID":"05ad455a-193e-433c-994e-44600efc8041","Type":"ContainerStarted","Data":"fad51ebdfafdb9f5bb591885d33d1d6d3e0a4bebc4528cee8c650c99ceef41c8"} Jul 02 19:02:53.801835 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.801773 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" event={"ID":"a3fd3a8f-67dd-41f4-b313-1c855db78fc3","Type":"ContainerStarted","Data":"216e1c13237956f049e5c3aaacc947eb15b42377e02840e96b158e5c93d062f9"} Jul 02 19:02:53.811901 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.811804 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2m8c5" event={"ID":"bb5fb82b-d3ee-4078-bd9b-391072dbb2f6","Type":"ContainerStarted","Data":"5bd8018668f8900e4695a9e1b25f19ece4779752b187e2d46a451b60ef12793d"} Jul 02 19:02:53.815741 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:53.815667 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vd52d" event={"ID":"57a35ed1-ba9b-41d9-bf9b-bd99874366c6","Type":"ContainerStarted","Data":"89d631f922ff7bfb14455acbc5f926ebbfa17d88b22ab4130a264acf0c5ee233"} Jul 02 19:02:54.226399 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:54.225614 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:02:54.226399 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:54.225661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs\") pod \"network-metrics-daemon-c2q69\" (UID: \"cf954a6b-7e4e-472a-8755-eafa3367d32c\") " pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:02:54.226399 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:54.225821 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:02:54.226399 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:54.225903 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs podName:cf954a6b-7e4e-472a-8755-eafa3367d32c nodeName:}" failed. No retries permitted until 2026-07-02 19:02:56.225883335 +0000 UTC m=+6.022753509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs") pod "network-metrics-daemon-c2q69" (UID: "cf954a6b-7e4e-472a-8755-eafa3367d32c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:02:54.226399 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:54.226313 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:02:54.226399 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:54.226364 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret podName:6f01d507-0625-477b-bf2f-7382a4c5bc9b nodeName:}" failed. No retries permitted until 2026-07-02 19:02:56.226348098 +0000 UTC m=+6.023218281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret") pod "global-pull-secret-syncer-t7n9j" (UID: "6f01d507-0625-477b-bf2f-7382a4c5bc9b") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:02:54.326805 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:54.326713 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4tw\" (UniqueName: \"kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw\") pod \"network-check-target-xmmlt\" (UID: \"0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48\") " pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:02:54.326998 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:54.326919 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:02:54.326998 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:54.326939 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:02:54.326998 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:54.326951 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hc4tw for pod openshift-network-diagnostics/network-check-target-xmmlt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:02:54.327159 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:54.327009 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw podName:0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48 nodeName:}" failed. No retries permitted until 2026-07-02 19:02:56.3269916 +0000 UTC m=+6.123861786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hc4tw" (UniqueName: "kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw") pod "network-check-target-xmmlt" (UID: "0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:02:54.641719 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:54.641598 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 18:57:52 +0000 UTC" deadline="2028-03-09 17:28:36.052994298 +0000 UTC" Jul 02 19:02:54.641719 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:54.641637 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14782h25m41.411362182s" Jul 02 19:02:54.761148 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:54.760290 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:02:54.761148 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:54.760450 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:02:54.761148 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:54.760865 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:02:54.761148 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:54.760960 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:02:54.761148 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:54.761028 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:02:54.761148 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:54.761098 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:02:54.832718 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:54.832681 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal" event={"ID":"857dad294ddde2e2b97208e4a9c2f3af","Type":"ContainerStarted","Data":"74e1353d3d8113161ebd240f80730833d73be7cbebef913f6364e6784fb3b570"} Jul 02 19:02:54.848007 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:54.847946 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-150.ec2.internal" podStartSLOduration=2.847924641 podStartE2EDuration="2.847924641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:02:54.847909041 +0000 UTC m=+4.644779421" watchObservedRunningTime="2026-07-02 19:02:54.847924641 +0000 UTC m=+4.644794833" Jul 02 19:02:56.244088 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:56.244045 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:02:56.244541 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:56.244098 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs\") pod \"network-metrics-daemon-c2q69\" (UID: \"cf954a6b-7e4e-472a-8755-eafa3367d32c\") " pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:02:56.244541 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:56.244264 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:02:56.244541 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:56.244346 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs podName:cf954a6b-7e4e-472a-8755-eafa3367d32c nodeName:}" failed. No retries permitted until 2026-07-02 19:03:00.244307669 +0000 UTC m=+10.041177853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs") pod "network-metrics-daemon-c2q69" (UID: "cf954a6b-7e4e-472a-8755-eafa3367d32c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:02:56.244768 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:56.244751 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:02:56.244820 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:56.244800 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret podName:6f01d507-0625-477b-bf2f-7382a4c5bc9b nodeName:}" failed. No retries permitted until 2026-07-02 19:03:00.24478625 +0000 UTC m=+10.041656422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret") pod "global-pull-secret-syncer-t7n9j" (UID: "6f01d507-0625-477b-bf2f-7382a4c5bc9b") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:02:56.345654 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:56.345025 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4tw\" (UniqueName: \"kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw\") pod \"network-check-target-xmmlt\" (UID: \"0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48\") " pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:02:56.345654 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:56.345232 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:02:56.345654 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:56.345252 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:02:56.345654 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:56.345264 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hc4tw for pod openshift-network-diagnostics/network-check-target-xmmlt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:02:56.345654 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:56.345323 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw podName:0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48 nodeName:}" failed. No retries permitted until 2026-07-02 19:03:00.345304273 +0000 UTC m=+10.142174461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hc4tw" (UniqueName: "kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw") pod "network-check-target-xmmlt" (UID: "0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:02:56.758088 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:56.758053 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:02:56.758248 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:56.758202 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:02:56.758291 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:56.758256 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:02:56.758352 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:56.758334 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:02:56.758512 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:56.758481 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:02:56.758606 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:56.758587 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:02:58.759980 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:58.759901 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:02:58.760354 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:58.760001 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:02:58.760392 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:58.760368 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:02:58.760478 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:58.760459 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:02:58.760529 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:02:58.760510 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:02:58.760582 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:02:58.760562 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:00.280972 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:00.280453 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:00.280972 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:00.280499 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs\") pod \"network-metrics-daemon-c2q69\" (UID: \"cf954a6b-7e4e-472a-8755-eafa3367d32c\") " pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:00.280972 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:00.280617 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:03:00.280972 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:00.280675 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs podName:cf954a6b-7e4e-472a-8755-eafa3367d32c nodeName:}" failed. No retries permitted until 2026-07-02 19:03:08.280656266 +0000 UTC m=+18.077526458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs") pod "network-metrics-daemon-c2q69" (UID: "cf954a6b-7e4e-472a-8755-eafa3367d32c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:03:00.280972 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:00.280955 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:03:00.281571 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:00.281013 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret podName:6f01d507-0625-477b-bf2f-7382a4c5bc9b nodeName:}" failed. No retries permitted until 2026-07-02 19:03:08.280998461 +0000 UTC m=+18.077868632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret") pod "global-pull-secret-syncer-t7n9j" (UID: "6f01d507-0625-477b-bf2f-7382a4c5bc9b") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:03:00.381207 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:00.381165 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4tw\" (UniqueName: \"kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw\") pod \"network-check-target-xmmlt\" (UID: \"0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48\") " pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:00.381390 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:00.381298 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:03:00.381390 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:00.381324 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:03:00.381390 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:00.381338 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hc4tw for pod openshift-network-diagnostics/network-check-target-xmmlt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:03:00.381557 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:00.381400 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw podName:0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48 nodeName:}" failed. No retries permitted until 2026-07-02 19:03:08.381380495 +0000 UTC m=+18.178250681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hc4tw" (UniqueName: "kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw") pod "network-check-target-xmmlt" (UID: "0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:03:00.760200 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:00.760117 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:00.760365 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:00.760233 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:00.760632 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:00.760612 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:00.760732 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:00.760705 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:00.760863 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:00.760764 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:00.760863 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:00.760856 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:02.760014 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:02.759952 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:02.760564 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:02.760093 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:02.760564 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:02.760338 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:02.760564 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:02.760424 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:02.760564 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:02.760455 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:02.760564 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:02.760531 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:04.758016 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:04.757975 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:04.758470 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:04.758119 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:04.758470 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:04.757980 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:04.758470 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:04.758204 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:04.758470 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:04.758214 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:04.758470 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:04.758312 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:06.758065 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:06.758034 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:06.758065 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:06.758068 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:06.758487 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:06.758042 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:06.758487 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:06.758159 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:06.758487 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:06.758256 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:06.758487 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:06.758323 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:08.340783 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:08.340704 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:08.340783 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:08.340742 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs\") pod \"network-metrics-daemon-c2q69\" (UID: \"cf954a6b-7e4e-472a-8755-eafa3367d32c\") " pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:08.341334 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:08.340890 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:03:08.341334 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:08.340954 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs podName:cf954a6b-7e4e-472a-8755-eafa3367d32c nodeName:}" failed. No retries permitted until 2026-07-02 19:03:24.34093383 +0000 UTC m=+34.137804004 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs") pod "network-metrics-daemon-c2q69" (UID: "cf954a6b-7e4e-472a-8755-eafa3367d32c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:03:08.341334 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:08.340961 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:03:08.341334 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:08.341019 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret podName:6f01d507-0625-477b-bf2f-7382a4c5bc9b nodeName:}" failed. No retries permitted until 2026-07-02 19:03:24.341002332 +0000 UTC m=+34.137872505 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret") pod "global-pull-secret-syncer-t7n9j" (UID: "6f01d507-0625-477b-bf2f-7382a4c5bc9b") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:03:08.441774 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:08.441736 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4tw\" (UniqueName: \"kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw\") pod \"network-check-target-xmmlt\" (UID: \"0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48\") " pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:08.441940 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:08.441877 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:03:08.441940 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:08.441899 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:03:08.441940 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:08.441911 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hc4tw for pod openshift-network-diagnostics/network-check-target-xmmlt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:03:08.442116 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:08.441968 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw podName:0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48 nodeName:}" failed. No retries permitted until 2026-07-02 19:03:24.441950369 +0000 UTC m=+34.238820537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hc4tw" (UniqueName: "kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw") pod "network-check-target-xmmlt" (UID: "0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:03:08.758265 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:08.758144 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:08.758265 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:08.758163 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:08.758265 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:08.758144 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:08.758514 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:08.758271 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:08.758514 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:08.758354 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:08.758514 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:08.758430 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:10.759165 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:10.759133 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:10.759707 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:10.759221 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:10.759707 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:10.759262 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:10.759707 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:10.759274 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:10.759707 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:10.759335 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:10.759707 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:10.759419 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:12.757778 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.757603 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:12.758460 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.757603 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:12.758460 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:12.757879 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:12.758460 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.757608 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:12.758460 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:12.757960 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:12.758460 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:12.758010 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:12.874600 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.874457 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c485v" event={"ID":"05ad455a-193e-433c-994e-44600efc8041","Type":"ContainerStarted","Data":"2415fd4c7f670f2eb1bc26c1cb10028aaf1957bf49d62ce2aab5f500b9cca084"} Jul 02 19:03:12.876231 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.876214 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:03:12.876509 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.876493 2565 generic.go:356] "Generic (PLEG): container finished" podID="a3fd3a8f-67dd-41f4-b313-1c855db78fc3" containerID="1a11ea1fa51ae93c04c81a391056fac73001717f0235b42eb737b68f3a9461c4" exitCode=1 Jul 02 19:03:12.876566 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.876549 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" event={"ID":"a3fd3a8f-67dd-41f4-b313-1c855db78fc3","Type":"ContainerStarted","Data":"4f5e027eb90149924c9e4d187772b3893580307de786cf7b9344b463f39711c7"} Jul 02 19:03:12.876619 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.876608 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" event={"ID":"a3fd3a8f-67dd-41f4-b313-1c855db78fc3","Type":"ContainerDied","Data":"1a11ea1fa51ae93c04c81a391056fac73001717f0235b42eb737b68f3a9461c4"} Jul 02 19:03:12.876667 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.876656 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" event={"ID":"a3fd3a8f-67dd-41f4-b313-1c855db78fc3","Type":"ContainerStarted","Data":"39b3d3628bc0b540dcd1be7baa291b28a59e4068d872c65f2f4a3f98a926b490"} Jul 02 19:03:12.877751 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.877733 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vd52d" event={"ID":"57a35ed1-ba9b-41d9-bf9b-bd99874366c6","Type":"ContainerStarted","Data":"80e3629b4f844e0a13cd35186c36ef18c4fe85273a8387d76fada4b33258fb2e"} Jul 02 19:03:12.878973 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.878953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" event={"ID":"73035ff3-c9bf-4f9e-9d60-40d7282f97c4","Type":"ContainerStarted","Data":"394841c2705b67a6ea2f3b204cb2ecfd4946aa1ab2693f58df5593523cc11b22"} Jul 02 19:03:12.880295 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.880277 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zxctb" event={"ID":"f583ae21-02a0-4d33-b57c-b8370bbaecdc","Type":"ContainerStarted","Data":"20c91bbb064139e12899dbb0f1b5855b5ac4c058a9d20e8e9ccf14aa4ddf29e4"} Jul 02 19:03:12.881598 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.881578 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab" containerID="dee61b7d6d2c92adaa6530ff46e52dac43d89aa14adff5091195a4a46dc3516e" exitCode=0 Jul 02 19:03:12.881668 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.881636 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdm4z" event={"ID":"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab","Type":"ContainerDied","Data":"dee61b7d6d2c92adaa6530ff46e52dac43d89aa14adff5091195a4a46dc3516e"} Jul 02 19:03:12.883109 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.883089 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rkjd9" event={"ID":"35b8716c-c781-4444-85ea-b906b33f52fb","Type":"ContainerStarted","Data":"d11c51ba2c129052806abfd4c3fa5cd3ace414fdd2d6bb9a40737f7a366f68f0"} Jul 02 19:03:12.884720 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.884688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" event={"ID":"562e2387-8a49-4d49-b44a-4006eb653991","Type":"ContainerStarted","Data":"93418c5fe84d304e76b9d19773f2fd0ffcdcd943aa8bc9f9de944cb0120d7ad1"} Jul 02 19:03:12.886036 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.886016 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-150.ec2.internal" event={"ID":"4dee40681f34b7bc84fa53a0b9970f9c","Type":"ContainerStarted","Data":"428152d767ee2a3b93396825da1de893a1187b7d950983f23efc0635fac73abd"} Jul 02 19:03:12.892131 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.892092 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c485v" podStartSLOduration=3.888985832 podStartE2EDuration="22.892080373s" totalImagesPullingTime="19.003094541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:02:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:12.891572154 +0000 UTC m=+22.688442344" watchObservedRunningTime="2026-07-02 19:03:12.892080373 +0000 UTC m=+22.688950582" Jul 02 19:03:12.908533 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.905485 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-150.ec2.internal" podStartSLOduration=20.905468305 podStartE2EDuration="20.905468305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:12.90478456 +0000 UTC m=+22.701654750" watchObservedRunningTime="2026-07-02 19:03:12.905468305 +0000 UTC m=+22.702338495" Jul 02 19:03:12.943152 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.943109 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zxctb" podStartSLOduration=3.898807606 podStartE2EDuration="22.943092316s" totalImagesPullingTime="19.04428471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:02:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:12.94262941 +0000 UTC m=+22.739499601" watchObservedRunningTime="2026-07-02 19:03:12.943092316 +0000 UTC m=+22.739962508" Jul 02 19:03:12.973086 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.973049 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vd52d" podStartSLOduration=4.009690212 podStartE2EDuration="22.973032929s" totalImagesPullingTime="18.963342717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:02:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:12.955819885 +0000 UTC m=+22.752690076" watchObservedRunningTime="2026-07-02 19:03:12.973032929 +0000 UTC m=+22.769903121" Jul 02 19:03:12.973190 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.973121 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rkjd9" podStartSLOduration=3.9638953 podStartE2EDuration="22.973112607s" totalImagesPullingTime="19.009217307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:02:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:12.972712607 +0000 UTC m=+22.769582810" watchObservedRunningTime="2026-07-02 19:03:12.973112607 +0000 UTC m=+22.769982800" Jul 02 19:03:12.989703 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:12.989670 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fp8qp" podStartSLOduration=2.977292699 podStartE2EDuration="21.989657609s" totalImagesPullingTime="19.01236491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:12.989348688 +0000 UTC m=+22.786218890" watchObservedRunningTime="2026-07-02 19:03:12.989657609 +0000 UTC m=+22.786527800" Jul 02 19:03:13.298717 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:13.298629 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:13.498335 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:13.498305 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:13.631910 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:13.631888 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 02 19:03:13.664806 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:13.664727 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T19:03:13.631905309Z","UUID":"bf481642-94c3-4610-bfcc-7f6bb7ce9d1b","Handler":null,"Name":"","Endpoint":""} Jul 02 19:03:13.666033 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:13.666015 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 02 19:03:13.666110 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:13.666049 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 02 19:03:13.699447 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:13.699398 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:13.800559 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:13.800525 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:13.890522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:13.890451 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:03:13.890879 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:13.890829 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" event={"ID":"a3fd3a8f-67dd-41f4-b313-1c855db78fc3","Type":"ContainerStarted","Data":"e6e76067900a585e233076d6a5c8fe44aa2546c787bcc2dc4cc6bbbe3e43aef8"} Jul 02 19:03:13.890999 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:13.890947 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" event={"ID":"a3fd3a8f-67dd-41f4-b313-1c855db78fc3","Type":"ContainerStarted","Data":"63e9413e274ab4fcda8be4603306e31cf455390b35c30be43fdc7fffec963196"} Jul 02 19:03:13.890999 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:13.890985 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" event={"ID":"a3fd3a8f-67dd-41f4-b313-1c855db78fc3","Type":"ContainerStarted","Data":"1e15cd27647e30910370c028d57b7fd355cd9ede69d86256762a84ea14adf29f"} Jul 02 19:03:13.892634 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:13.892607 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" event={"ID":"73035ff3-c9bf-4f9e-9d60-40d7282f97c4","Type":"ContainerStarted","Data":"aa20c941f01401a387346acc31f3526029d375fc7a9f730aa13f1b3e7dd41b1a"} Jul 02 19:03:13.900043 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:13.900021 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:14.091705 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:14.091676 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:14.757879 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:14.757832 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:14.758016 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:14.757832 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:14.758016 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:14.757976 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:14.758016 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:14.757981 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:14.758197 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:14.758076 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:14.758197 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:14.758155 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:14.896021 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:14.895984 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2m8c5" event={"ID":"bb5fb82b-d3ee-4078-bd9b-391072dbb2f6","Type":"ContainerStarted","Data":"4e04e568669d8cb2dca03a536a81a6d0e090bc0c628df8b9091110dd81597ecc"} Jul 02 19:03:14.897967 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:14.897945 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" event={"ID":"73035ff3-c9bf-4f9e-9d60-40d7282f97c4","Type":"ContainerStarted","Data":"4cbd4ff9489de5592db46ea9845df64bd9044f08abc1f1d75c89836960e2aa8f"} Jul 02 19:03:14.909622 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:14.909586 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2m8c5" podStartSLOduration=5.903364412 podStartE2EDuration="24.909573681s" totalImagesPullingTime="19.006209269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:02:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:14.909379689 +0000 UTC m=+24.706249879" watchObservedRunningTime="2026-07-02 19:03:14.909573681 +0000 UTC m=+24.706443871" Jul 02 19:03:14.925133 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:14.925096 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8ctnq" podStartSLOduration=2.7770540009999998 podStartE2EDuration="23.925083148s" totalImagesPullingTime="21.148029147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:02:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:14.924973298 +0000 UTC m=+24.721843490" watchObservedRunningTime="2026-07-02 19:03:14.925083148 +0000 UTC m=+24.721953352" Jul 02 19:03:15.095695 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:15.095669 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:15.494756 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:15.494549 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:15.694616 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:15.694585 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:15.894095 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:15.894013 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:15.904141 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:15.904120 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:03:15.904600 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:15.904571 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" event={"ID":"a3fd3a8f-67dd-41f4-b313-1c855db78fc3","Type":"ContainerStarted","Data":"82fda5e6f655e326d7659e3a41ca283e92b2c318d630a4707f1578addb766a03"} Jul 02 19:03:16.058907 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:16.058858 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c485v" Jul 02 19:03:16.059699 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:16.059681 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c485v" Jul 02 19:03:16.303572 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:16.303484 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:16.695395 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:16.695314 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:16.758298 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:16.758267 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:16.758451 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:16.758266 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:16.758451 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:16.758369 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:16.758572 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:16.758480 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:16.758572 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:16.758274 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:16.758651 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:16.758568 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:16.906919 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:16.906888 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c485v" Jul 02 19:03:16.907412 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:16.907388 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c485v" Jul 02 19:03:17.293034 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:17.292998 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:17.358634 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:17.358531 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:17.524377 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:17.524115 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:17.824159 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:17.824063 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:17.912436 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:17.912411 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:03:17.913216 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:17.912881 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" event={"ID":"a3fd3a8f-67dd-41f4-b313-1c855db78fc3","Type":"ContainerStarted","Data":"aed950e994a54d08ea14d4213b448811f4b213c2737f5fcccc4e755a839d5a73"} Jul 02 19:03:17.913368 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:17.913352 2565 scope.go:117] "RemoveContainer" containerID="1a11ea1fa51ae93c04c81a391056fac73001717f0235b42eb737b68f3a9461c4" Jul 02 19:03:18.098780 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:18.098694 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:18.639141 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:18.639112 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:18.757633 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:18.757601 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:18.757633 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:18.757620 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:18.757872 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:18.757760 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:18.757872 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:18.757815 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:18.757872 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:18.757836 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:18.757999 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:18.757916 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:19.918585 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:19.918549 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab" containerID="9f7fec1bb8da6d4e4ced87d3cc4f0f8b0e9e5ce1145d1f8384c9c1800d6fe7fb" exitCode=0 Jul 02 19:03:19.919040 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:19.918619 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdm4z" event={"ID":"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab","Type":"ContainerDied","Data":"9f7fec1bb8da6d4e4ced87d3cc4f0f8b0e9e5ce1145d1f8384c9c1800d6fe7fb"} Jul 02 19:03:19.922251 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:19.922233 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:03:19.922646 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:19.922628 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" event={"ID":"a3fd3a8f-67dd-41f4-b313-1c855db78fc3","Type":"ContainerStarted","Data":"d26e9b8a3fbe555ac26be2bdd335042f6dd6ebd631425f65e02b43ae7f460b62"} Jul 02 19:03:19.923057 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:19.923033 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:03:19.923152 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:19.923115 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:03:19.923213 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:19.923167 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:03:19.938348 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:19.938328 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:03:19.938472 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:19.938459 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:03:19.962785 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:19.962742 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" podStartSLOduration=9.628060039 podStartE2EDuration="28.962731085s" totalImagesPullingTime="19.334671046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:19.961557215 +0000 UTC m=+29.758427400" watchObservedRunningTime="2026-07-02 19:03:19.962731085 +0000 UTC m=+29.759601276" Jul 02 19:03:20.760871 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:20.759731 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:20.760871 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:20.759906 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:20.760871 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:20.760340 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:20.760871 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:20.760455 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:20.760871 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:20.760527 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:20.760871 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:20.760633 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:21.928671 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:21.928632 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab" containerID="dd59c46f7c53df24a6d854d1e6879db5d3234733b1845e033931009ce98889ea" exitCode=0 Jul 02 19:03:21.929104 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:21.928726 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdm4z" event={"ID":"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab","Type":"ContainerDied","Data":"dd59c46f7c53df24a6d854d1e6879db5d3234733b1845e033931009ce98889ea"} Jul 02 19:03:22.757647 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:22.757614 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:22.757647 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:22.757639 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:22.757907 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:22.757616 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:22.757907 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:22.757726 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:22.757907 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:22.757797 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:22.758004 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:22.757901 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:23.935531 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:23.935492 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab" containerID="1a0f457b15f80a3ef8e203144ce5ab666d8127a296120bcca1f65a61bbbc8b57" exitCode=0 Jul 02 19:03:23.935914 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:23.935541 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdm4z" event={"ID":"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab","Type":"ContainerDied","Data":"1a0f457b15f80a3ef8e203144ce5ab666d8127a296120bcca1f65a61bbbc8b57"} Jul 02 19:03:24.358939 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:24.358911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:24.359104 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:24.358950 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs\") pod \"network-metrics-daemon-c2q69\" (UID: \"cf954a6b-7e4e-472a-8755-eafa3367d32c\") " pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:24.359104 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:24.359046 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:03:24.359104 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:24.359047 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:03:24.359104 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:24.359100 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret podName:6f01d507-0625-477b-bf2f-7382a4c5bc9b nodeName:}" failed. No retries permitted until 2026-07-02 19:03:56.359083884 +0000 UTC m=+66.155954053 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret") pod "global-pull-secret-syncer-t7n9j" (UID: "6f01d507-0625-477b-bf2f-7382a4c5bc9b") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:03:24.359242 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:24.359114 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs podName:cf954a6b-7e4e-472a-8755-eafa3367d32c nodeName:}" failed. No retries permitted until 2026-07-02 19:03:56.359107185 +0000 UTC m=+66.155977354 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs") pod "network-metrics-daemon-c2q69" (UID: "cf954a6b-7e4e-472a-8755-eafa3367d32c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:03:24.459783 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:24.459753 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4tw\" (UniqueName: \"kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw\") pod \"network-check-target-xmmlt\" (UID: \"0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48\") " pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:24.459939 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:24.459881 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:03:24.459939 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:24.459897 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:03:24.459939 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:24.459905 2565 projected.go:196] Error preparing data for projected volume kube-api-access-hc4tw for pod openshift-network-diagnostics/network-check-target-xmmlt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:03:24.460038 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:24.459946 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw podName:0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48 nodeName:}" failed. No retries permitted until 2026-07-02 19:03:56.459933744 +0000 UTC m=+66.256803913 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hc4tw" (UniqueName: "kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw") pod "network-check-target-xmmlt" (UID: "0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:03:24.758401 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:24.757995 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:24.758401 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:24.758017 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:24.758401 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:24.758127 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:24.758401 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:24.758185 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:24.758401 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:24.758286 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:24.758401 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:24.758360 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:26.231237 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:26.231205 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:26.321354 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:26.321321 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:26.544873 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:26.544761 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:26.682241 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:26.682206 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:26.758186 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:26.758153 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:26.758186 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:26.758179 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:26.758431 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:26.758154 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:26.758431 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:26.758279 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:26.758431 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:26.758365 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:26.758780 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:26.758457 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:26.803036 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:26.802968 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:27.176036 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:27.175959 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:27.271626 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:27.271586 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:27.374154 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:27.374075 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:27.570222 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:27.570186 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:27.627302 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:27.627229 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:27.714205 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:27.714057 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:27.816511 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:27.816481 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:27.847041 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:27.846957 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t7n9j"] Jul 02 19:03:27.847167 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:27.847088 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:27.847227 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:27.847197 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:27.849701 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:27.849621 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c2q69"] Jul 02 19:03:27.849880 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:27.849762 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:27.849973 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:27.849889 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:27.850412 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:27.850391 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xmmlt"] Jul 02 19:03:27.850559 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:27.850483 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:27.850623 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:27.850575 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:27.916125 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:27.916080 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:28.180993 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:28.180922 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:28.274789 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:28.274752 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:29.136327 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:29.136286 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:29.330504 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:29.330475 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:29.467253 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:29.467229 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:29.594130 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:29.594099 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:29.693438 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:29.693411 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:29.757976 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:29.757951 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:29.757976 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:29.757981 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:29.758133 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:29.757995 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:29.758133 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:29.758092 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:29.758218 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:29.758188 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:29.758266 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:29.758242 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:29.952479 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:29.952424 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab" containerID="5d987963ed090f42234e7871ab850a26de70b58bf71b29b1471bee5db34f1210" exitCode=0 Jul 02 19:03:29.952479 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:29.952464 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdm4z" event={"ID":"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab","Type":"ContainerDied","Data":"5d987963ed090f42234e7871ab850a26de70b58bf71b29b1471bee5db34f1210"} Jul 02 19:03:30.155532 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:30.155511 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:30.256497 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:30.256469 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:30.957006 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:30.956975 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab" containerID="8a5393e6d14cd65ed81a780bf0ac07fbb3a525ac9efbdbb224aa13f813352915" exitCode=0 Jul 02 19:03:30.957705 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:30.957017 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdm4z" event={"ID":"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab","Type":"ContainerDied","Data":"8a5393e6d14cd65ed81a780bf0ac07fbb3a525ac9efbdbb224aa13f813352915"} Jul 02 19:03:31.758105 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:31.758077 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:31.758105 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:31.758104 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:31.758298 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:31.758078 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:31.758298 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:31.758183 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c2q69" podUID="cf954a6b-7e4e-472a-8755-eafa3367d32c" Jul 02 19:03:31.758298 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:31.758259 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t7n9j" podUID="6f01d507-0625-477b-bf2f-7382a4c5bc9b" Jul 02 19:03:31.758390 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:31.758298 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xmmlt" podUID="0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48" Jul 02 19:03:31.962337 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:31.962299 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kdm4z" event={"ID":"7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab","Type":"ContainerStarted","Data":"a231f791e40b9888b5f72914f4a146d1f6d7c778cf25ad3c7280a58611cc0ba9"} Jul 02 19:03:31.984017 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:31.983973 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kdm4z" podStartSLOduration=8.654884259 podStartE2EDuration="41.983960376s" totalImagesPullingTime="33.329076117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:02:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:31.982720594 +0000 UTC m=+41.779590797" watchObservedRunningTime="2026-07-02 19:03:31.983960376 +0000 UTC m=+41.780830567" Jul 02 19:03:32.108954 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.108875 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:32.172738 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.172709 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:32.324706 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.324669 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:32.391775 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.391704 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:32.440618 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.440596 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-150.ec2.internal" event="NodeReady" Jul 02 19:03:32.440722 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.440699 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 02 19:03:32.499451 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.499423 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rscwv"] Jul 02 19:03:32.522662 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.522639 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hfq7v"] Jul 02 19:03:32.522802 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.522775 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rscwv" Jul 02 19:03:32.526490 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.526468 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 02 19:03:32.526605 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.526509 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 02 19:03:32.526605 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.526545 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 02 19:03:32.526719 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.526702 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cf2kh\"" Jul 02 19:03:32.540212 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.540188 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rscwv"] Jul 02 19:03:32.540311 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.540225 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d65jh"] Jul 02 19:03:32.540365 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.540334 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.543146 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.543128 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-psrnn\"" Jul 02 19:03:32.543466 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.543449 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 02 19:03:32.544548 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.544057 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 02 19:03:32.544548 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.544243 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 02 19:03:32.544548 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.544521 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 02 19:03:32.554103 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.554085 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hfq7v"] Jul 02 19:03:32.554179 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.554118 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d65jh"] Jul 02 19:03:32.554213 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.554191 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:32.558738 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.558724 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g97vz\"" Jul 02 19:03:32.559001 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.558989 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 02 19:03:32.563854 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.563826 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 02 19:03:32.621388 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.621353 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28fbcbc4-81c8-4e67-b448-e912dd00eb27-cert\") pod \"ingress-canary-rscwv\" (UID: \"28fbcbc4-81c8-4e67-b448-e912dd00eb27\") " pod="openshift-ingress-canary/ingress-canary-rscwv" Jul 02 19:03:32.621471 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.621413 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n4dl\" (UniqueName: \"kubernetes.io/projected/28fbcbc4-81c8-4e67-b448-e912dd00eb27-kube-api-access-9n4dl\") pod \"ingress-canary-rscwv\" (UID: \"28fbcbc4-81c8-4e67-b448-e912dd00eb27\") " pod="openshift-ingress-canary/ingress-canary-rscwv" Jul 02 19:03:32.721976 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.721919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56-tmp-dir\") pod \"dns-default-d65jh\" (UID: \"6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56\") " pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:32.721976 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.721949 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28fbcbc4-81c8-4e67-b448-e912dd00eb27-cert\") pod \"ingress-canary-rscwv\" (UID: \"28fbcbc4-81c8-4e67-b448-e912dd00eb27\") " pod="openshift-ingress-canary/ingress-canary-rscwv" Jul 02 19:03:32.721976 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.721967 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f63be4f-6f08-489d-bc63-b33240cf694b-data-volume\") pod \"insights-runtime-extractor-hfq7v\" (UID: \"0f63be4f-6f08-489d-bc63-b33240cf694b\") " pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.722173 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.721984 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f63be4f-6f08-489d-bc63-b33240cf694b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hfq7v\" (UID: \"0f63be4f-6f08-489d-bc63-b33240cf694b\") " pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.722173 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.722029 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktdrh\" (UniqueName: \"kubernetes.io/projected/6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56-kube-api-access-ktdrh\") pod \"dns-default-d65jh\" (UID: \"6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56\") " pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:32.722173 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.722066 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n4dl\" (UniqueName: \"kubernetes.io/projected/28fbcbc4-81c8-4e67-b448-e912dd00eb27-kube-api-access-9n4dl\") pod \"ingress-canary-rscwv\" (UID: \"28fbcbc4-81c8-4e67-b448-e912dd00eb27\") " pod="openshift-ingress-canary/ingress-canary-rscwv" Jul 02 19:03:32.722173 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.722084 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56-config-volume\") pod \"dns-default-d65jh\" (UID: \"6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56\") " pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:32.722173 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.722100 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gws4g\" (UniqueName: \"kubernetes.io/projected/0f63be4f-6f08-489d-bc63-b33240cf694b-kube-api-access-gws4g\") pod \"insights-runtime-extractor-hfq7v\" (UID: \"0f63be4f-6f08-489d-bc63-b33240cf694b\") " pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.722173 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.722121 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56-metrics-tls\") pod \"dns-default-d65jh\" (UID: \"6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56\") " pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:32.722173 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.722136 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f63be4f-6f08-489d-bc63-b33240cf694b-crio-socket\") pod \"insights-runtime-extractor-hfq7v\" (UID: \"0f63be4f-6f08-489d-bc63-b33240cf694b\") " pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.722173 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.722175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f63be4f-6f08-489d-bc63-b33240cf694b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hfq7v\" (UID: \"0f63be4f-6f08-489d-bc63-b33240cf694b\") " pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.726438 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.726418 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28fbcbc4-81c8-4e67-b448-e912dd00eb27-cert\") pod \"ingress-canary-rscwv\" (UID: \"28fbcbc4-81c8-4e67-b448-e912dd00eb27\") " pod="openshift-ingress-canary/ingress-canary-rscwv" Jul 02 19:03:32.731433 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.731415 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n4dl\" (UniqueName: \"kubernetes.io/projected/28fbcbc4-81c8-4e67-b448-e912dd00eb27-kube-api-access-9n4dl\") pod \"ingress-canary-rscwv\" (UID: \"28fbcbc4-81c8-4e67-b448-e912dd00eb27\") " pod="openshift-ingress-canary/ingress-canary-rscwv" Jul 02 19:03:32.755657 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.755633 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:32.823118 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.823092 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f63be4f-6f08-489d-bc63-b33240cf694b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hfq7v\" (UID: \"0f63be4f-6f08-489d-bc63-b33240cf694b\") " pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.823232 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.823136 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56-tmp-dir\") pod \"dns-default-d65jh\" (UID: \"6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56\") " pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:32.823232 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.823154 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f63be4f-6f08-489d-bc63-b33240cf694b-data-volume\") pod \"insights-runtime-extractor-hfq7v\" (UID: \"0f63be4f-6f08-489d-bc63-b33240cf694b\") " pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.823232 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.823169 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f63be4f-6f08-489d-bc63-b33240cf694b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hfq7v\" (UID: \"0f63be4f-6f08-489d-bc63-b33240cf694b\") " pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.823232 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.823194 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktdrh\" (UniqueName: \"kubernetes.io/projected/6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56-kube-api-access-ktdrh\") pod \"dns-default-d65jh\" (UID: \"6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56\") " pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:32.823232 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.823225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56-config-volume\") pod \"dns-default-d65jh\" (UID: \"6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56\") " pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:32.823444 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.823240 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gws4g\" (UniqueName: \"kubernetes.io/projected/0f63be4f-6f08-489d-bc63-b33240cf694b-kube-api-access-gws4g\") pod \"insights-runtime-extractor-hfq7v\" (UID: \"0f63be4f-6f08-489d-bc63-b33240cf694b\") " pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.823444 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.823257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56-metrics-tls\") pod \"dns-default-d65jh\" (UID: \"6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56\") " pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:32.823444 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.823276 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f63be4f-6f08-489d-bc63-b33240cf694b-crio-socket\") pod \"insights-runtime-extractor-hfq7v\" (UID: \"0f63be4f-6f08-489d-bc63-b33240cf694b\") " pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.823577 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.823526 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56-tmp-dir\") pod \"dns-default-d65jh\" (UID: \"6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56\") " pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:32.823639 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.823574 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f63be4f-6f08-489d-bc63-b33240cf694b-data-volume\") pod \"insights-runtime-extractor-hfq7v\" (UID: \"0f63be4f-6f08-489d-bc63-b33240cf694b\") " pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.823716 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.823691 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f63be4f-6f08-489d-bc63-b33240cf694b-crio-socket\") pod \"insights-runtime-extractor-hfq7v\" (UID: \"0f63be4f-6f08-489d-bc63-b33240cf694b\") " pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.823790 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.823774 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f63be4f-6f08-489d-bc63-b33240cf694b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hfq7v\" (UID: \"0f63be4f-6f08-489d-bc63-b33240cf694b\") " pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.823944 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.823923 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56-config-volume\") pod \"dns-default-d65jh\" (UID: \"6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56\") " pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:32.825671 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.825652 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f63be4f-6f08-489d-bc63-b33240cf694b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hfq7v\" (UID: \"0f63be4f-6f08-489d-bc63-b33240cf694b\") " pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.825741 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.825689 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56-metrics-tls\") pod \"dns-default-d65jh\" (UID: \"6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56\") " pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:32.831911 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.831893 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rscwv" Jul 02 19:03:32.832220 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.832201 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gws4g\" (UniqueName: \"kubernetes.io/projected/0f63be4f-6f08-489d-bc63-b33240cf694b-kube-api-access-gws4g\") pod \"insights-runtime-extractor-hfq7v\" (UID: \"0f63be4f-6f08-489d-bc63-b33240cf694b\") " pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.832292 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.832277 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktdrh\" (UniqueName: \"kubernetes.io/projected/6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56-kube-api-access-ktdrh\") pod \"dns-default-d65jh\" (UID: \"6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56\") " pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:32.849173 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.849155 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hfq7v" Jul 02 19:03:32.854638 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.854620 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:32.861618 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:32.861592 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:33.065035 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:33.065000 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d65jh"] Jul 02 19:03:33.071119 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:03:33.071092 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce2b18f_f9a2_46aa_bd2a_6f6aaccddb56.slice/crio-0a1be6c3b6381c8efa4e201a635a7646dee502e7456af27120d4165e43f7f4c2 WatchSource:0}: Error finding container 0a1be6c3b6381c8efa4e201a635a7646dee502e7456af27120d4165e43f7f4c2: Status 404 returned error can't find the container with id 0a1be6c3b6381c8efa4e201a635a7646dee502e7456af27120d4165e43f7f4c2 Jul 02 19:03:33.079939 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:33.079914 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rscwv"] Jul 02 19:03:33.081794 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:33.081770 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hfq7v"] Jul 02 19:03:33.084766 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:03:33.084744 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f63be4f_6f08_489d_bc63_b33240cf694b.slice/crio-99f349d21af31d950976756f4e9376382f265f548c30c0559db5a50c5484e479 WatchSource:0}: Error finding container 99f349d21af31d950976756f4e9376382f265f548c30c0559db5a50c5484e479: Status 404 returned error can't find the container with id 99f349d21af31d950976756f4e9376382f265f548c30c0559db5a50c5484e479 Jul 02 19:03:33.085810 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:03:33.085791 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fbcbc4_81c8_4e67_b448_e912dd00eb27.slice/crio-80e6e43f12c718c186cff959178f8ff9c822bdac96174c8f172e774f09f6f57e WatchSource:0}: Error finding container 80e6e43f12c718c186cff959178f8ff9c822bdac96174c8f172e774f09f6f57e: Status 404 returned error can't find the container with id 80e6e43f12c718c186cff959178f8ff9c822bdac96174c8f172e774f09f6f57e Jul 02 19:03:33.758293 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:33.758257 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:33.758465 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:33.758386 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:33.758465 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:33.758421 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:33.761745 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:33.761723 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 19:03:33.763616 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:33.763588 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 19:03:33.763869 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:33.763836 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 19:03:33.764122 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:33.764104 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bswjt\"" Jul 02 19:03:33.764349 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:33.764330 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xfkkw\"" Jul 02 19:03:33.764539 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:33.764518 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 19:03:33.970917 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:33.970882 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d65jh" event={"ID":"6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56","Type":"ContainerStarted","Data":"0a1be6c3b6381c8efa4e201a635a7646dee502e7456af27120d4165e43f7f4c2"} Jul 02 19:03:33.972929 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:33.972833 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hfq7v" event={"ID":"0f63be4f-6f08-489d-bc63-b33240cf694b","Type":"ContainerStarted","Data":"99f349d21af31d950976756f4e9376382f265f548c30c0559db5a50c5484e479"} Jul 02 19:03:33.974615 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:33.974557 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rscwv" event={"ID":"28fbcbc4-81c8-4e67-b448-e912dd00eb27","Type":"ContainerStarted","Data":"80e6e43f12c718c186cff959178f8ff9c822bdac96174c8f172e774f09f6f57e"} Jul 02 19:03:35.752474 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.752288 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dcb57cbbf-jkrpc"] Jul 02 19:03:35.762920 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.762899 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:35.766134 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.765704 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dcb57cbbf-jkrpc"] Jul 02 19:03:35.766134 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.765790 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hzr27\"" Jul 02 19:03:35.766134 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.765976 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 02 19:03:35.766134 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.766100 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 02 19:03:35.767480 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.766662 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 02 19:03:35.767480 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.766668 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 02 19:03:35.767480 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.766685 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 02 19:03:35.767480 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.766725 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 02 19:03:35.767480 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.766666 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 02 19:03:35.948419 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.945696 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-serving-cert\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:35.948419 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.945770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-service-ca\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:35.948419 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.945866 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-config\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:35.948419 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.945901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-oauth-serving-cert\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:35.948419 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.945961 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-oauth-config\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:35.948419 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:35.945998 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zb99\" (UniqueName: \"kubernetes.io/projected/27802ccf-b4fe-4d73-a774-91fc256bd7ea-kube-api-access-4zb99\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:36.046783 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:36.046700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-oauth-config\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:36.046783 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:36.046751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zb99\" (UniqueName: \"kubernetes.io/projected/27802ccf-b4fe-4d73-a774-91fc256bd7ea-kube-api-access-4zb99\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:36.047019 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:36.046809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-serving-cert\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:36.047019 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:36.046955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-service-ca\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:36.047120 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:36.047044 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-config\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:36.047120 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:36.047079 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-oauth-serving-cert\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:36.047678 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:36.047649 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-service-ca\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:36.047792 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:36.047747 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-config\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:36.047836 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:36.047823 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-oauth-serving-cert\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:36.050428 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:36.050407 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-serving-cert\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:36.050556 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:36.050537 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-oauth-config\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:36.055206 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:36.055186 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zb99\" (UniqueName: \"kubernetes.io/projected/27802ccf-b4fe-4d73-a774-91fc256bd7ea-kube-api-access-4zb99\") pod \"console-6dcb57cbbf-jkrpc\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:36.074265 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:36.074237 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:38.074324 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:38.074288 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:38.175020 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:38.174987 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:38.413997 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:38.413575 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:38.485475 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:38.484229 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dcb57cbbf-jkrpc"] Jul 02 19:03:38.490288 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:03:38.490252 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27802ccf_b4fe_4d73_a774_91fc256bd7ea.slice/crio-1f76d59199c6a282eab396f8a2c101913af60c5b6499ddd65725826c5a19f8cf WatchSource:0}: Error finding container 1f76d59199c6a282eab396f8a2c101913af60c5b6499ddd65725826c5a19f8cf: Status 404 returned error can't find the container with id 1f76d59199c6a282eab396f8a2c101913af60c5b6499ddd65725826c5a19f8cf Jul 02 19:03:38.512252 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:38.512186 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:38.603805 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:38.603712 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:38.848879 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:38.848828 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:38.944578 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:38.944505 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:38.988720 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:38.988687 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rscwv" event={"ID":"28fbcbc4-81c8-4e67-b448-e912dd00eb27","Type":"ContainerStarted","Data":"40c69d225172e4b90c2bc0004e738795e1d9bea6fcaa839d9f8e4ab5617935d1"} Jul 02 19:03:38.990073 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:38.990044 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcb57cbbf-jkrpc" event={"ID":"27802ccf-b4fe-4d73-a774-91fc256bd7ea","Type":"ContainerStarted","Data":"1f76d59199c6a282eab396f8a2c101913af60c5b6499ddd65725826c5a19f8cf"} Jul 02 19:03:38.991638 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:38.991610 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d65jh" event={"ID":"6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56","Type":"ContainerStarted","Data":"7c976ae7b807a1cb3bf76d0e80ff65907edd9f5b36ab3e3c9225edd820b02d0d"} Jul 02 19:03:38.991732 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:38.991673 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d65jh" event={"ID":"6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56","Type":"ContainerStarted","Data":"148de289aaab9946a7ec0fa00223767113e24aff7b4b57af249d14bcdadff809"} Jul 02 19:03:38.991732 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:38.991705 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:38.993063 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:38.993034 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hfq7v" event={"ID":"0f63be4f-6f08-489d-bc63-b33240cf694b","Type":"ContainerStarted","Data":"83e46b634a3787aa97f3495dbf1e6a3444234bbef916e19ad75c6e9d7e0c5665"} Jul 02 19:03:39.005619 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:39.005583 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rscwv" podStartSLOduration=1.760765981 podStartE2EDuration="7.005570546s" totalImagesPullingTime="5.244804565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:39.004390422 +0000 UTC m=+48.801260613" watchObservedRunningTime="2026-07-02 19:03:39.005570546 +0000 UTC m=+48.802440737" Jul 02 19:03:39.026035 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:39.025998 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d65jh" podStartSLOduration=1.766985904 podStartE2EDuration="7.025984258s" totalImagesPullingTime="5.258998354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:03:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:39.024749438 +0000 UTC m=+48.821619632" watchObservedRunningTime="2026-07-02 19:03:39.025984258 +0000 UTC m=+48.822854449" Jul 02 19:03:39.997466 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:39.997422 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hfq7v" event={"ID":"0f63be4f-6f08-489d-bc63-b33240cf694b","Type":"ContainerStarted","Data":"15bdce002e8bcaa23c6eb4c2fa594fe8b8dd8b793a29b4176005ed347063a366"} Jul 02 19:03:40.743323 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:40.743251 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:40.807542 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:40.807508 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:41.670208 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.670174 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:41.709184 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.709146 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7"] Jul 02 19:03:41.711176 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.711152 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" Jul 02 19:03:41.714109 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.714085 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-hz9c7\"" Jul 02 19:03:41.715733 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.715401 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 02 19:03:41.715733 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.715407 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 02 19:03:41.715733 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.715440 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 02 19:03:41.715733 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.715470 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 02 19:03:41.715733 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.715425 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 02 19:03:41.727763 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.727741 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7"] Jul 02 19:03:41.760768 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.760734 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:41.789005 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.788972 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1072a934-656e-4676-98eb-1d4c432e733c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-p2kw7\" (UID: \"1072a934-656e-4676-98eb-1d4c432e733c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" Jul 02 19:03:41.789130 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.789101 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1072a934-656e-4676-98eb-1d4c432e733c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-p2kw7\" (UID: \"1072a934-656e-4676-98eb-1d4c432e733c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" Jul 02 19:03:41.789185 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.789126 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzq25\" (UniqueName: \"kubernetes.io/projected/1072a934-656e-4676-98eb-1d4c432e733c-kube-api-access-wzq25\") pod \"prometheus-operator-567f7b8b7c-p2kw7\" (UID: \"1072a934-656e-4676-98eb-1d4c432e733c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" Jul 02 19:03:41.789238 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.789196 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1072a934-656e-4676-98eb-1d4c432e733c-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-p2kw7\" (UID: \"1072a934-656e-4676-98eb-1d4c432e733c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" Jul 02 19:03:41.890567 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.890528 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1072a934-656e-4676-98eb-1d4c432e733c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-p2kw7\" (UID: \"1072a934-656e-4676-98eb-1d4c432e733c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" Jul 02 19:03:41.890737 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.890578 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzq25\" (UniqueName: \"kubernetes.io/projected/1072a934-656e-4676-98eb-1d4c432e733c-kube-api-access-wzq25\") pod \"prometheus-operator-567f7b8b7c-p2kw7\" (UID: \"1072a934-656e-4676-98eb-1d4c432e733c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" Jul 02 19:03:41.890737 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.890631 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1072a934-656e-4676-98eb-1d4c432e733c-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-p2kw7\" (UID: \"1072a934-656e-4676-98eb-1d4c432e733c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" Jul 02 19:03:41.890737 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.890664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1072a934-656e-4676-98eb-1d4c432e733c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-p2kw7\" (UID: \"1072a934-656e-4676-98eb-1d4c432e733c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" Jul 02 19:03:41.890932 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:41.890796 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 02 19:03:41.890932 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:41.890886 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1072a934-656e-4676-98eb-1d4c432e733c-prometheus-operator-tls podName:1072a934-656e-4676-98eb-1d4c432e733c nodeName:}" failed. No retries permitted until 2026-07-02 19:03:42.390866771 +0000 UTC m=+52.187736942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/1072a934-656e-4676-98eb-1d4c432e733c-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-p2kw7" (UID: "1072a934-656e-4676-98eb-1d4c432e733c") : secret "prometheus-operator-tls" not found Jul 02 19:03:41.891501 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.891476 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1072a934-656e-4676-98eb-1d4c432e733c-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-p2kw7\" (UID: \"1072a934-656e-4676-98eb-1d4c432e733c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" Jul 02 19:03:41.894642 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.894618 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1072a934-656e-4676-98eb-1d4c432e733c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-p2kw7\" (UID: \"1072a934-656e-4676-98eb-1d4c432e733c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" Jul 02 19:03:41.902108 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:41.902085 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzq25\" (UniqueName: \"kubernetes.io/projected/1072a934-656e-4676-98eb-1d4c432e733c-kube-api-access-wzq25\") pod \"prometheus-operator-567f7b8b7c-p2kw7\" (UID: \"1072a934-656e-4676-98eb-1d4c432e733c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" Jul 02 19:03:42.010346 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:42.010300 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:42.394189 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:42.394101 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1072a934-656e-4676-98eb-1d4c432e733c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-p2kw7\" (UID: \"1072a934-656e-4676-98eb-1d4c432e733c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" Jul 02 19:03:42.396898 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:42.396868 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1072a934-656e-4676-98eb-1d4c432e733c-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-p2kw7\" (UID: \"1072a934-656e-4676-98eb-1d4c432e733c\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" Jul 02 19:03:42.621502 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:42.621465 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" Jul 02 19:03:42.770353 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:42.770314 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7"] Jul 02 19:03:42.774767 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:03:42.774746 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1072a934_656e_4676_98eb_1d4c432e733c.slice/crio-9ca0257c73cceb7b4d6467d8fc8542ba30e73b56e970f267ffde857e2c709e48 WatchSource:0}: Error finding container 9ca0257c73cceb7b4d6467d8fc8542ba30e73b56e970f267ffde857e2c709e48: Status 404 returned error can't find the container with id 9ca0257c73cceb7b4d6467d8fc8542ba30e73b56e970f267ffde857e2c709e48 Jul 02 19:03:42.948530 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:42.948452 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-576f7d85f-st92t"] Jul 02 19:03:42.951698 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:42.951677 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:42.961317 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:42.961299 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 02 19:03:42.965094 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:42.965072 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-576f7d85f-st92t"] Jul 02 19:03:42.999699 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:42.999678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d427d2f0-abca-4e47-b38b-c1f809edd977-console-serving-cert\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:42.999827 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:42.999718 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d427d2f0-abca-4e47-b38b-c1f809edd977-console-oauth-config\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:42.999827 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:42.999787 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9zsc\" (UniqueName: \"kubernetes.io/projected/d427d2f0-abca-4e47-b38b-c1f809edd977-kube-api-access-s9zsc\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:42.999827 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:42.999824 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-service-ca\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:42.999978 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:42.999944 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-trusted-ca-bundle\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:42.999978 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:42.999967 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-oauth-serving-cert\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.000052 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.000008 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-console-config\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.009393 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.009371 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcb57cbbf-jkrpc" event={"ID":"27802ccf-b4fe-4d73-a774-91fc256bd7ea","Type":"ContainerStarted","Data":"7712051998bc753bcde6007df5b5dee53a5a8d2186fb6323dedcae21d05e7974"} Jul 02 19:03:43.010525 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.010505 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" event={"ID":"1072a934-656e-4676-98eb-1d4c432e733c","Type":"ContainerStarted","Data":"9ca0257c73cceb7b4d6467d8fc8542ba30e73b56e970f267ffde857e2c709e48"} Jul 02 19:03:43.012412 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.012393 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hfq7v" event={"ID":"0f63be4f-6f08-489d-bc63-b33240cf694b","Type":"ContainerStarted","Data":"41b4db31ae6ce8e19e6f033be94fbbbab58141e24df0c01b8f1b4539b8af50bb"} Jul 02 19:03:43.029157 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.029114 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dcb57cbbf-jkrpc" podStartSLOduration=3.881048626 podStartE2EDuration="8.029102565s" totalImagesPullingTime="4.148053939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:43.028832027 +0000 UTC m=+52.825702217" watchObservedRunningTime="2026-07-02 19:03:43.029102565 +0000 UTC m=+52.825972756" Jul 02 19:03:43.048730 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.048688 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hfq7v" podStartSLOduration=1.6417574099999999 podStartE2EDuration="11.048676021s" totalImagesPullingTime="9.406918611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:03:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:43.047955888 +0000 UTC m=+52.844826088" watchObservedRunningTime="2026-07-02 19:03:43.048676021 +0000 UTC m=+52.845546220" Jul 02 19:03:43.100487 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.100464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d427d2f0-abca-4e47-b38b-c1f809edd977-console-serving-cert\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.100629 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.100611 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d427d2f0-abca-4e47-b38b-c1f809edd977-console-oauth-config\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.100752 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.100738 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9zsc\" (UniqueName: \"kubernetes.io/projected/d427d2f0-abca-4e47-b38b-c1f809edd977-kube-api-access-s9zsc\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.100807 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.100783 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-service-ca\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.100889 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.100874 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-trusted-ca-bundle\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.100956 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.100907 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-oauth-serving-cert\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.100956 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.100939 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-console-config\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.101567 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.101543 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-console-config\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.101775 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.101754 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-service-ca\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.101936 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.101919 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-oauth-serving-cert\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.102035 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.102021 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-trusted-ca-bundle\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.104643 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.104621 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d427d2f0-abca-4e47-b38b-c1f809edd977-console-oauth-config\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.104743 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.104723 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d427d2f0-abca-4e47-b38b-c1f809edd977-console-serving-cert\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.110147 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.110121 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9zsc\" (UniqueName: \"kubernetes.io/projected/d427d2f0-abca-4e47-b38b-c1f809edd977-kube-api-access-s9zsc\") pod \"console-576f7d85f-st92t\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.261080 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.261055 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:43.389942 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.389919 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-576f7d85f-st92t"] Jul 02 19:03:43.391336 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:03:43.391304 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd427d2f0_abca_4e47_b38b_c1f809edd977.slice/crio-2b11e05cba19c2de5a11aaff38d1e94f578bc89a45ff6fb89678064e474d3c64 WatchSource:0}: Error finding container 2b11e05cba19c2de5a11aaff38d1e94f578bc89a45ff6fb89678064e474d3c64: Status 404 returned error can't find the container with id 2b11e05cba19c2de5a11aaff38d1e94f578bc89a45ff6fb89678064e474d3c64 Jul 02 19:03:43.951857 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:43.951806 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:44.007103 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:44.007068 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:44.017918 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:44.017399 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-576f7d85f-st92t" event={"ID":"d427d2f0-abca-4e47-b38b-c1f809edd977","Type":"ContainerStarted","Data":"0babc0c924bdda9ec6b2a3125a97e6e6c7c10afdde16d8a78c519e8f801a965f"} Jul 02 19:03:44.017918 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:44.017458 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-576f7d85f-st92t" event={"ID":"d427d2f0-abca-4e47-b38b-c1f809edd977","Type":"ContainerStarted","Data":"2b11e05cba19c2de5a11aaff38d1e94f578bc89a45ff6fb89678064e474d3c64"} Jul 02 19:03:44.035941 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:44.035898 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-576f7d85f-st92t" podStartSLOduration=2.035881467 podStartE2EDuration="2.035881467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:44.034888028 +0000 UTC m=+53.831758222" watchObservedRunningTime="2026-07-02 19:03:44.035881467 +0000 UTC m=+53.832751636" Jul 02 19:03:44.516880 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:44.516823 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:44.617998 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:44.617962 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:44.675621 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:44.675576 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:45.022514 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:45.022481 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" event={"ID":"1072a934-656e-4676-98eb-1d4c432e733c","Type":"ContainerStarted","Data":"f78ea2104d75a85fd03e14a4ed2f367b17da12534d719c0df6922c8708b59b4a"} Jul 02 19:03:45.022923 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:45.022538 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" event={"ID":"1072a934-656e-4676-98eb-1d4c432e733c","Type":"ContainerStarted","Data":"ce87f8a9cc196d070ca39ab4e09083f976d4e8c0f539e6124e9da4f9bdafb762"} Jul 02 19:03:45.096572 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:45.096496 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:45.197106 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:45.197072 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:45.966240 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:45.966197 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:46.021182 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:46.021150 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:46.041815 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:46.041774 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-p2kw7" podStartSLOduration=2.998038931 podStartE2EDuration="5.041757205s" totalImagesPullingTime="2.043718274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:03:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:46.041321085 +0000 UTC m=+55.838191287" watchObservedRunningTime="2026-07-02 19:03:46.041757205 +0000 UTC m=+55.838627397" Jul 02 19:03:46.075145 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:46.075118 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:46.075270 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:46.075177 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:03:46.076467 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:46.076444 2565 patch_prober.go:28] interesting pod/console-6dcb57cbbf-jkrpc container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" start-of-body= Jul 02 19:03:46.076575 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:46.076485 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6dcb57cbbf-jkrpc" podUID="27802ccf-b4fe-4d73-a774-91fc256bd7ea" containerName="console" probeResult="failure" output="Get \"https://10.134.0.9:8443/health\": dial tcp 10.134.0.9:8443: connect: connection refused" Jul 02 19:03:47.820100 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:47.820059 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:48.058203 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.058158 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:48.110226 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.110159 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n"] Jul 02 19:03:48.130710 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.130689 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n"] Jul 02 19:03:48.130818 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.130782 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" Jul 02 19:03:48.133804 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.133784 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 02 19:03:48.133804 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.133793 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qlt4x\"" Jul 02 19:03:48.134110 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.133828 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 02 19:03:48.134110 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.134068 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lr94p"] Jul 02 19:03:48.146605 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.146587 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.150126 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.150103 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 02 19:03:48.150234 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.150176 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2g2r7\"" Jul 02 19:03:48.150647 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.150633 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 02 19:03:48.150958 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.150938 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 02 19:03:48.193058 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.193038 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:48.239110 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.239079 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-root\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.239210 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.239112 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-node-exporter-textfile\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.239210 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.239135 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-node-exporter-accelerators-collector-config\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.239210 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.239193 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqnlp\" (UniqueName: \"kubernetes.io/projected/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-kube-api-access-qqnlp\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.239307 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.239244 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d036447-e4c8-4d76-a475-d159cba51f95-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-rr86n\" (UID: \"6d036447-e4c8-4d76-a475-d159cba51f95\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" Jul 02 19:03:48.239307 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.239286 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-node-exporter-wtmp\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.239307 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.239302 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcwl7\" (UniqueName: \"kubernetes.io/projected/6d036447-e4c8-4d76-a475-d159cba51f95-kube-api-access-qcwl7\") pod \"openshift-state-metrics-54cf96dfcf-rr86n\" (UID: \"6d036447-e4c8-4d76-a475-d159cba51f95\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" Jul 02 19:03:48.239394 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.239344 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d036447-e4c8-4d76-a475-d159cba51f95-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rr86n\" (UID: \"6d036447-e4c8-4d76-a475-d159cba51f95\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" Jul 02 19:03:48.239394 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.239379 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-metrics-client-ca\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.239451 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.239430 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d036447-e4c8-4d76-a475-d159cba51f95-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-rr86n\" (UID: \"6d036447-e4c8-4d76-a475-d159cba51f95\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" Jul 02 19:03:48.239480 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.239449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-node-exporter-tls\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.239480 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.239465 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.239544 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.239481 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-sys\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.340521 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.340491 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d036447-e4c8-4d76-a475-d159cba51f95-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-rr86n\" (UID: \"6d036447-e4c8-4d76-a475-d159cba51f95\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" Jul 02 19:03:48.340651 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.340531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-node-exporter-tls\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.340651 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.340549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.340651 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.340566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-sys\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.340651 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.340588 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-root\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.340651 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.340612 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-node-exporter-textfile\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.340651 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.340635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-node-exporter-accelerators-collector-config\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.340986 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.340650 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-sys\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.340986 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.340675 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-root\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.340986 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.340658 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqnlp\" (UniqueName: \"kubernetes.io/projected/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-kube-api-access-qqnlp\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.340986 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.340766 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d036447-e4c8-4d76-a475-d159cba51f95-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-rr86n\" (UID: \"6d036447-e4c8-4d76-a475-d159cba51f95\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" Jul 02 19:03:48.340986 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.340867 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-node-exporter-wtmp\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.340986 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.340900 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcwl7\" (UniqueName: \"kubernetes.io/projected/6d036447-e4c8-4d76-a475-d159cba51f95-kube-api-access-qcwl7\") pod \"openshift-state-metrics-54cf96dfcf-rr86n\" (UID: \"6d036447-e4c8-4d76-a475-d159cba51f95\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" Jul 02 19:03:48.340986 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.340949 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d036447-e4c8-4d76-a475-d159cba51f95-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rr86n\" (UID: \"6d036447-e4c8-4d76-a475-d159cba51f95\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" Jul 02 19:03:48.340986 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.340974 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-metrics-client-ca\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.341348 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.341068 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-node-exporter-textfile\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.341399 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.341371 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-node-exporter-accelerators-collector-config\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.341458 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.341415 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-node-exporter-wtmp\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.341507 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:48.341461 2565 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 02 19:03:48.341566 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:03:48.341518 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d036447-e4c8-4d76-a475-d159cba51f95-openshift-state-metrics-tls podName:6d036447-e4c8-4d76-a475-d159cba51f95 nodeName:}" failed. No retries permitted until 2026-07-02 19:03:48.841501744 +0000 UTC m=+58.638371913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6d036447-e4c8-4d76-a475-d159cba51f95-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-rr86n" (UID: "6d036447-e4c8-4d76-a475-d159cba51f95") : secret "openshift-state-metrics-tls" not found Jul 02 19:03:48.341566 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.341543 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-metrics-client-ca\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.341644 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.341565 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6d036447-e4c8-4d76-a475-d159cba51f95-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-rr86n\" (UID: \"6d036447-e4c8-4d76-a475-d159cba51f95\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" Jul 02 19:03:48.344169 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.344147 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.344432 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.344415 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6d036447-e4c8-4d76-a475-d159cba51f95-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-rr86n\" (UID: \"6d036447-e4c8-4d76-a475-d159cba51f95\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" Jul 02 19:03:48.344507 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.344450 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-node-exporter-tls\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.346048 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.346027 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:48.350893 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.350872 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcwl7\" (UniqueName: \"kubernetes.io/projected/6d036447-e4c8-4d76-a475-d159cba51f95-kube-api-access-qcwl7\") pod \"openshift-state-metrics-54cf96dfcf-rr86n\" (UID: \"6d036447-e4c8-4d76-a475-d159cba51f95\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" Jul 02 19:03:48.350984 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.350944 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqnlp\" (UniqueName: \"kubernetes.io/projected/b182dd57-e318-4ba4-8ceb-8e9fb7d85406-kube-api-access-qqnlp\") pod \"node-exporter-lr94p\" (UID: \"b182dd57-e318-4ba4-8ceb-8e9fb7d85406\") " pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.445097 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.445015 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:48.454794 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.454766 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lr94p" Jul 02 19:03:48.497647 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:03:48.496959 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb182dd57_e318_4ba4_8ceb_8e9fb7d85406.slice/crio-3e7bdf75ee498f730029bad49945f6ff683254b7f74f76f36e9707f46ac94113 WatchSource:0}: Error finding container 3e7bdf75ee498f730029bad49945f6ff683254b7f74f76f36e9707f46ac94113: Status 404 returned error can't find the container with id 3e7bdf75ee498f730029bad49945f6ff683254b7f74f76f36e9707f46ac94113 Jul 02 19:03:48.689893 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.689856 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:48.828924 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.828891 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:48.844521 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.844489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d036447-e4c8-4d76-a475-d159cba51f95-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rr86n\" (UID: \"6d036447-e4c8-4d76-a475-d159cba51f95\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" Jul 02 19:03:48.846895 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:48.846879 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6d036447-e4c8-4d76-a475-d159cba51f95-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rr86n\" (UID: \"6d036447-e4c8-4d76-a475-d159cba51f95\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" Jul 02 19:03:49.000213 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:49.000182 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d65jh" Jul 02 19:03:49.039428 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:49.039303 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" Jul 02 19:03:49.042319 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:49.040748 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lr94p" event={"ID":"b182dd57-e318-4ba4-8ceb-8e9fb7d85406","Type":"ContainerStarted","Data":"3e7bdf75ee498f730029bad49945f6ff683254b7f74f76f36e9707f46ac94113"} Jul 02 19:03:49.065233 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:49.065168 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:49.159903 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:49.159873 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:49.198030 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:49.197999 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n"] Jul 02 19:03:49.200709 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:03:49.200682 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d036447_e4c8_4d76_a475_d159cba51f95.slice/crio-e1fe6b1a35e5e0d925849c81999b6436b43b83a25569284481bb57f04b6accb8 WatchSource:0}: Error finding container e1fe6b1a35e5e0d925849c81999b6436b43b83a25569284481bb57f04b6accb8: Status 404 returned error can't find the container with id e1fe6b1a35e5e0d925849c81999b6436b43b83a25569284481bb57f04b6accb8 Jul 02 19:03:49.287993 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:49.287966 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:50.048078 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.048039 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" event={"ID":"6d036447-e4c8-4d76-a475-d159cba51f95","Type":"ContainerStarted","Data":"0cf5eee0bcc6d06b8da412777633e013d23eabf202750c08c5d8361e1ef94ae5"} Jul 02 19:03:50.048431 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.048101 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" event={"ID":"6d036447-e4c8-4d76-a475-d159cba51f95","Type":"ContainerStarted","Data":"bdea825900fbd2592970c4bcac32c9ab2156641fea1fa99b371773e3df422028"} Jul 02 19:03:50.048431 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.048122 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" event={"ID":"6d036447-e4c8-4d76-a475-d159cba51f95","Type":"ContainerStarted","Data":"e1fe6b1a35e5e0d925849c81999b6436b43b83a25569284481bb57f04b6accb8"} Jul 02 19:03:50.073364 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.072135 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-69c68d759d-mmtpp"] Jul 02 19:03:50.102215 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.102191 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69c68d759d-mmtpp"] Jul 02 19:03:50.102347 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.102315 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.105073 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.105038 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 02 19:03:50.105201 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.105075 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 02 19:03:50.105201 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.105110 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-49k9502474v62\"" Jul 02 19:03:50.105201 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.105038 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 02 19:03:50.105201 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.105152 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 02 19:03:50.105391 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.105224 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 02 19:03:50.105391 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.105260 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mftzc\"" Jul 02 19:03:50.156718 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.156691 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-thanos-querier-tls\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.156831 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.156755 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.156831 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.156790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.156987 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.156905 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj28f\" (UniqueName: \"kubernetes.io/projected/e409ca8c-556c-445a-bd68-822247bbd65f-kube-api-access-lj28f\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.156987 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.156949 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.156987 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.156977 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e409ca8c-556c-445a-bd68-822247bbd65f-metrics-client-ca\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.157126 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.157019 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-grpc-tls\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.157126 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.157056 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.258227 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.258200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-thanos-querier-tls\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.258324 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.258278 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.258324 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.258317 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.258433 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.258353 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj28f\" (UniqueName: \"kubernetes.io/projected/e409ca8c-556c-445a-bd68-822247bbd65f-kube-api-access-lj28f\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.258433 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.258394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.258433 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.258422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e409ca8c-556c-445a-bd68-822247bbd65f-metrics-client-ca\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.258616 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.258468 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-grpc-tls\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.258616 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.258507 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.259550 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.259488 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e409ca8c-556c-445a-bd68-822247bbd65f-metrics-client-ca\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.261677 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.261631 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-thanos-querier-tls\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.261770 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.261674 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.261995 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.261967 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.262049 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.262020 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.262049 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.262041 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.262684 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.262664 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e409ca8c-556c-445a-bd68-822247bbd65f-secret-grpc-tls\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.269232 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.269211 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj28f\" (UniqueName: \"kubernetes.io/projected/e409ca8c-556c-445a-bd68-822247bbd65f-kube-api-access-lj28f\") pod \"thanos-querier-69c68d759d-mmtpp\" (UID: \"e409ca8c-556c-445a-bd68-822247bbd65f\") " pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.411720 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.411691 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:50.729078 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.729043 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69c68d759d-mmtpp"] Jul 02 19:03:50.732423 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:03:50.732398 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode409ca8c_556c_445a_bd68_822247bbd65f.slice/crio-9eb6cff45f3ede003e3322fbd43071e39a2cca668a789488766d9da943ff0225 WatchSource:0}: Error finding container 9eb6cff45f3ede003e3322fbd43071e39a2cca668a789488766d9da943ff0225: Status 404 returned error can't find the container with id 9eb6cff45f3ede003e3322fbd43071e39a2cca668a789488766d9da943ff0225 Jul 02 19:03:50.995110 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:50.995025 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:51.048452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:51.048421 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:51.052692 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:51.052662 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" event={"ID":"6d036447-e4c8-4d76-a475-d159cba51f95","Type":"ContainerStarted","Data":"7f3cc492ac84decfcd9f1ea97a8848298e1da8fb26cb12d642777f923414b829"} Jul 02 19:03:51.053717 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:51.053693 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" event={"ID":"e409ca8c-556c-445a-bd68-822247bbd65f","Type":"ContainerStarted","Data":"9eb6cff45f3ede003e3322fbd43071e39a2cca668a789488766d9da943ff0225"} Jul 02 19:03:51.055033 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:51.055008 2565 generic.go:356] "Generic (PLEG): container finished" podID="b182dd57-e318-4ba4-8ceb-8e9fb7d85406" containerID="16d54d185c610945f060a2c84d54762d04bdc68a1ff90ea6a4c1b1a797c8d0c4" exitCode=0 Jul 02 19:03:51.055138 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:51.055049 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lr94p" event={"ID":"b182dd57-e318-4ba4-8ceb-8e9fb7d85406","Type":"ContainerDied","Data":"16d54d185c610945f060a2c84d54762d04bdc68a1ff90ea6a4c1b1a797c8d0c4"} Jul 02 19:03:51.068923 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:51.068877 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rr86n" podStartSLOduration=1.9014906900000001 podStartE2EDuration="3.068863742s" totalImagesPullingTime="1.167373052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:51.068736291 +0000 UTC m=+60.865606484" watchObservedRunningTime="2026-07-02 19:03:51.068863742 +0000 UTC m=+60.865733927" Jul 02 19:03:51.122419 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:51.122392 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:51.222271 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:51.222241 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:51.453116 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:51.453083 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:51.538270 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:51.538243 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:51.621430 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:51.621371 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:51.941399 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:51.941368 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wx9nx" Jul 02 19:03:52.060221 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.060181 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lr94p" event={"ID":"b182dd57-e318-4ba4-8ceb-8e9fb7d85406","Type":"ContainerStarted","Data":"49bf7c311c245237f3afbee4ac769927c0065156b184edc7acbe77dc6e522b99"} Jul 02 19:03:52.060621 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.060263 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lr94p" event={"ID":"b182dd57-e318-4ba4-8ceb-8e9fb7d85406","Type":"ContainerStarted","Data":"c820ff8653026f53d14d45456efa3802eac1c7725f2e56b6875028b0e9d7d582"} Jul 02 19:03:52.085252 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.085209 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lr94p" podStartSLOduration=2.33459793 podStartE2EDuration="4.085194473s" totalImagesPullingTime="1.662852012s" totalInitContainerRuntime="87.744531ms" isStatefulPod=true podCreationTimestamp="2026-07-02 19:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:52.084150885 +0000 UTC m=+61.881021076" watchObservedRunningTime="2026-07-02 19:03:52.085194473 +0000 UTC m=+61.882064667" Jul 02 19:03:52.120577 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.120544 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:52.224553 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.224462 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:52.278301 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.278272 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:52.646085 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.646051 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:52.746599 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.746565 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:52.852292 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.852258 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:52.862480 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.862448 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dcb57cbbf-jkrpc"] Jul 02 19:03:52.891499 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.891470 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8598fddfc9-cm9bn"] Jul 02 19:03:52.893692 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.893674 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:52.908017 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.907992 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8598fddfc9-cm9bn"] Jul 02 19:03:52.929455 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.929424 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:52.980891 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.980861 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg926\" (UniqueName: \"kubernetes.io/projected/580ec438-54e1-4767-8f05-55cec36ea8fa-kube-api-access-hg926\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:52.981035 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.980927 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-oauth-serving-cert\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:52.981035 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.980985 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-service-ca\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:52.981035 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.981011 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/580ec438-54e1-4767-8f05-55cec36ea8fa-console-serving-cert\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:52.981194 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.981066 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/580ec438-54e1-4767-8f05-55cec36ea8fa-console-oauth-config\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:52.981194 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.981106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-trusted-ca-bundle\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:52.981194 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:52.981140 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-console-config\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.082458 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.082428 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg926\" (UniqueName: \"kubernetes.io/projected/580ec438-54e1-4767-8f05-55cec36ea8fa-kube-api-access-hg926\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.082968 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.082480 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-oauth-serving-cert\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.082968 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.082529 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-service-ca\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.082968 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.082546 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/580ec438-54e1-4767-8f05-55cec36ea8fa-console-serving-cert\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.082968 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.082578 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/580ec438-54e1-4767-8f05-55cec36ea8fa-console-oauth-config\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.082968 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.082818 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-trusted-ca-bundle\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.083249 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.082996 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-console-config\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.083497 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.083473 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-service-ca\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.083762 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.083698 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-trusted-ca-bundle\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.083762 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.083751 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-oauth-serving-cert\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.084046 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.084027 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-console-config\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.086117 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.086099 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/580ec438-54e1-4767-8f05-55cec36ea8fa-console-serving-cert\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.086223 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.086133 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/580ec438-54e1-4767-8f05-55cec36ea8fa-console-oauth-config\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.093312 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.093289 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg926\" (UniqueName: \"kubernetes.io/projected/580ec438-54e1-4767-8f05-55cec36ea8fa-kube-api-access-hg926\") pod \"console-8598fddfc9-cm9bn\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.125502 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.125473 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:53.204559 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.204470 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:03:53.261966 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.261937 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:53.262102 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.261993 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:03:53.263639 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.263596 2565 patch_prober.go:28] interesting pod/console-576f7d85f-st92t container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.11:8443/health\": dial tcp 10.134.0.11:8443: connect: connection refused" start-of-body= Jul 02 19:03:53.263749 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.263654 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-576f7d85f-st92t" podUID="d427d2f0-abca-4e47-b38b-c1f809edd977" containerName="console" probeResult="failure" output="Get \"https://10.134.0.11:8443/health\": dial tcp 10.134.0.11:8443: connect: connection refused" Jul 02 19:03:53.496393 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.496365 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8598fddfc9-cm9bn"] Jul 02 19:03:53.499488 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:03:53.499463 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580ec438_54e1_4767_8f05_55cec36ea8fa.slice/crio-8254f377f55a4c07442d8535fc15c07591a7eccfbb0c4f1723d560c9809c04be WatchSource:0}: Error finding container 8254f377f55a4c07442d8535fc15c07591a7eccfbb0c4f1723d560c9809c04be: Status 404 returned error can't find the container with id 8254f377f55a4c07442d8535fc15c07591a7eccfbb0c4f1723d560c9809c04be Jul 02 19:03:53.822681 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.822651 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:53.889358 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:53.889330 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:54.075455 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.075355 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" event={"ID":"e409ca8c-556c-445a-bd68-822247bbd65f","Type":"ContainerStarted","Data":"a8ac416006baeba9766619c0df07495b73edf1a8ceaedeae51baf453a672eb9b"} Jul 02 19:03:54.075632 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.075474 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" event={"ID":"e409ca8c-556c-445a-bd68-822247bbd65f","Type":"ContainerStarted","Data":"dea49f414e4d5a888e149ae9f2421973be06534104a2b3826b5fc4d8ccff4e03"} Jul 02 19:03:54.075632 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.075524 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" event={"ID":"e409ca8c-556c-445a-bd68-822247bbd65f","Type":"ContainerStarted","Data":"3009896b739fcff425478cf8db2aa3cc5d0e8af83895823bf11f8f7b2db2bfce"} Jul 02 19:03:54.077013 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.076985 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8598fddfc9-cm9bn" event={"ID":"580ec438-54e1-4767-8f05-55cec36ea8fa","Type":"ContainerStarted","Data":"f787168787b1fbb50fb9b4ad1551f9e841018f52bc0914c0567872591f782616"} Jul 02 19:03:54.077193 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.077177 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8598fddfc9-cm9bn" event={"ID":"580ec438-54e1-4767-8f05-55cec36ea8fa","Type":"ContainerStarted","Data":"8254f377f55a4c07442d8535fc15c07591a7eccfbb0c4f1723d560c9809c04be"} Jul 02 19:03:54.095060 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.095008 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8598fddfc9-cm9bn" podStartSLOduration=2.09498866 podStartE2EDuration="2.09498866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:54.093464185 +0000 UTC m=+63.890334390" watchObservedRunningTime="2026-07-02 19:03:54.09498866 +0000 UTC m=+63.891858851" Jul 02 19:03:54.262194 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.262155 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 19:03:54.266210 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.264918 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.267351 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.267324 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:54.267758 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.267739 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 02 19:03:54.267881 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.267801 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pk4g2\"" Jul 02 19:03:54.269115 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.269094 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 02 19:03:54.269232 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.269094 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 02 19:03:54.269232 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.269143 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 02 19:03:54.269342 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.269228 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 02 19:03:54.270119 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.270058 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 02 19:03:54.270119 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.270100 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 02 19:03:54.270261 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.270152 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 02 19:03:54.270261 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.270101 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 02 19:03:54.270261 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.270197 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c0plndh4iapsl\"" Jul 02 19:03:54.270536 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.270518 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 02 19:03:54.270536 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.270531 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 02 19:03:54.270671 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.270657 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 02 19:03:54.272665 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.271161 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 02 19:03:54.273931 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.273911 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 02 19:03:54.275962 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.275941 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 02 19:03:54.286147 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.285611 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 19:03:54.394686 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.394585 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.394686 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.394618 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.394686 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.394668 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.394686 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.394688 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395037 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.394764 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4e118da-3b87-429d-9353-37ba79dc3205-config-out\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395037 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.394827 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395037 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.394874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395037 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.394918 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4e118da-3b87-429d-9353-37ba79dc3205-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395037 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.394937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395037 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.394957 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395037 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.394980 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395388 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.395065 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395388 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.395096 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-config\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395388 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.395118 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395388 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.395143 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d4e118da-3b87-429d-9353-37ba79dc3205-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395388 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.395168 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395388 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.395194 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k2rc\" (UniqueName: \"kubernetes.io/projected/d4e118da-3b87-429d-9353-37ba79dc3205-kube-api-access-7k2rc\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395388 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.395218 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-web-config\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395388 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.395252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.395388 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.395271 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.495857 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.495799 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.495857 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.495862 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496084 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.495917 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496084 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.495944 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496084 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.495988 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4e118da-3b87-429d-9353-37ba79dc3205-config-out\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496084 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496010 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496084 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496036 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496084 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4e118da-3b87-429d-9353-37ba79dc3205-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496372 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496094 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496372 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496372 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496148 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496372 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496187 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496372 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496216 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-config\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496372 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496242 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496372 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d4e118da-3b87-429d-9353-37ba79dc3205-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496372 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496303 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496372 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496334 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k2rc\" (UniqueName: \"kubernetes.io/projected/d4e118da-3b87-429d-9353-37ba79dc3205-kube-api-access-7k2rc\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496372 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496361 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-web-config\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496866 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496400 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496866 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496429 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.496866 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496619 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.497028 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.496883 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.499329 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.497533 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.499329 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.498013 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d4e118da-3b87-429d-9353-37ba79dc3205-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.499329 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.498238 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.499329 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.498543 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.499329 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.499046 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.500085 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.500048 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.500449 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.500426 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d4e118da-3b87-429d-9353-37ba79dc3205-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.501217 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.501194 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.501505 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.501478 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.502645 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.502619 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.503623 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.503549 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.504299 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.504249 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d4e118da-3b87-429d-9353-37ba79dc3205-config-out\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.504632 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.504609 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.505011 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.504943 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-config\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.505236 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.505219 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d4e118da-3b87-429d-9353-37ba79dc3205-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.505358 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.505315 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-web-config\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.506069 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.506050 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d4e118da-3b87-429d-9353-37ba79dc3205-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.507264 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.507225 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k2rc\" (UniqueName: \"kubernetes.io/projected/d4e118da-3b87-429d-9353-37ba79dc3205-kube-api-access-7k2rc\") pod \"prometheus-k8s-0\" (UID: \"d4e118da-3b87-429d-9353-37ba79dc3205\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.578568 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.578541 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:03:54.617961 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.617933 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:54.719120 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.719089 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:54.742100 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:54.741894 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 02 19:03:54.744180 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:03:54.744156 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e118da_3b87_429d_9353_37ba79dc3205.slice/crio-24d1d50a242549f777b860471b23248fe29e3d44ddea5f77e10d20778de5b312 WatchSource:0}: Error finding container 24d1d50a242549f777b860471b23248fe29e3d44ddea5f77e10d20778de5b312: Status 404 returned error can't find the container with id 24d1d50a242549f777b860471b23248fe29e3d44ddea5f77e10d20778de5b312 Jul 02 19:03:55.083328 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:55.083289 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" event={"ID":"e409ca8c-556c-445a-bd68-822247bbd65f","Type":"ContainerStarted","Data":"84ba5be7cb16f9847866fcfe59f7ddc29d092ae242a937b79b663abe0ead7c2d"} Jul 02 19:03:55.083506 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:55.083396 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" event={"ID":"e409ca8c-556c-445a-bd68-822247bbd65f","Type":"ContainerStarted","Data":"547282224ed49a56ebdb8c70153411df470287237377a03c89d05f345beaf709"} Jul 02 19:03:55.083506 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:55.083441 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" event={"ID":"e409ca8c-556c-445a-bd68-822247bbd65f","Type":"ContainerStarted","Data":"ade70bf589b5041d5a2e98651b654aaa0c5e48fd8fb3c336b8f710531c12b426"} Jul 02 19:03:55.083506 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:55.083482 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:03:55.084673 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:55.084651 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e118da-3b87-429d-9353-37ba79dc3205","Type":"ContainerStarted","Data":"24d1d50a242549f777b860471b23248fe29e3d44ddea5f77e10d20778de5b312"} Jul 02 19:03:55.106980 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:55.106939 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" podStartSLOduration=1.494463428 podStartE2EDuration="5.106924451s" totalImagesPullingTime="3.612461023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:03:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:55.105709635 +0000 UTC m=+64.902579847" watchObservedRunningTime="2026-07-02 19:03:55.106924451 +0000 UTC m=+64.903794641" Jul 02 19:03:55.621778 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:55.621737 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:55.720178 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:55.720137 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:55.779532 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:55.779503 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:56.089787 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.089752 2565 generic.go:356] "Generic (PLEG): container finished" podID="d4e118da-3b87-429d-9353-37ba79dc3205" containerID="b99263163ef937250068b518d279290fe696d28a1a7c97ed540e1c49ee1605ae" exitCode=0 Jul 02 19:03:56.089936 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.089793 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e118da-3b87-429d-9353-37ba79dc3205","Type":"ContainerDied","Data":"b99263163ef937250068b518d279290fe696d28a1a7c97ed540e1c49ee1605ae"} Jul 02 19:03:56.158163 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.158131 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:56.258350 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.258320 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:56.415652 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.415618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:56.415652 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.415655 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs\") pod \"network-metrics-daemon-c2q69\" (UID: \"cf954a6b-7e4e-472a-8755-eafa3367d32c\") " pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:56.418416 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.418401 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 19:03:56.418453 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.418402 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 19:03:56.428780 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.428751 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f01d507-0625-477b-bf2f-7382a4c5bc9b-original-pull-secret\") pod \"global-pull-secret-syncer-t7n9j\" (UID: \"6f01d507-0625-477b-bf2f-7382a4c5bc9b\") " pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:56.428780 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.428775 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf954a6b-7e4e-472a-8755-eafa3367d32c-metrics-certs\") pod \"network-metrics-daemon-c2q69\" (UID: \"cf954a6b-7e4e-472a-8755-eafa3367d32c\") " pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:56.516224 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.516161 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc4tw\" (UniqueName: \"kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw\") pod \"network-check-target-xmmlt\" (UID: \"0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48\") " pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:56.518943 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.518926 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 19:03:56.529973 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.529952 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 19:03:56.540411 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.540387 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc4tw\" (UniqueName: \"kubernetes.io/projected/0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48-kube-api-access-hc4tw\") pod \"network-check-target-xmmlt\" (UID: \"0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48\") " pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:56.575003 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.574985 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t7n9j" Jul 02 19:03:56.588423 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.588406 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bswjt\"" Jul 02 19:03:56.595566 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.595551 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:03:56.595792 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.595762 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xfkkw\"" Jul 02 19:03:56.603589 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.603568 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c2q69" Jul 02 19:03:56.735299 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.735121 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t7n9j"] Jul 02 19:03:56.740090 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:03:56.740054 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f01d507_0625_477b_bf2f_7382a4c5bc9b.slice/crio-d812dea1544e3890e517a29e215c10cdbeab6600cf5adeb5ee3d9494a8e9c4b9 WatchSource:0}: Error finding container d812dea1544e3890e517a29e215c10cdbeab6600cf5adeb5ee3d9494a8e9c4b9: Status 404 returned error can't find the container with id d812dea1544e3890e517a29e215c10cdbeab6600cf5adeb5ee3d9494a8e9c4b9 Jul 02 19:03:56.764109 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.764018 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c2q69"] Jul 02 19:03:56.767682 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:03:56.767624 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf954a6b_7e4e_472a_8755_eafa3367d32c.slice/crio-4ec6c541d2894100c57a64c646fcc7c0cb96647435d6b6280abda1a290fae11d WatchSource:0}: Error finding container 4ec6c541d2894100c57a64c646fcc7c0cb96647435d6b6280abda1a290fae11d: Status 404 returned error can't find the container with id 4ec6c541d2894100c57a64c646fcc7c0cb96647435d6b6280abda1a290fae11d Jul 02 19:03:56.780963 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:56.780917 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xmmlt"] Jul 02 19:03:56.782645 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:03:56.782627 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2d2be1_7e16_4d04_b95f_65c3ab6f9d48.slice/crio-34d5c5e5d1dd5734e52d73daf1236a97b1a0e3d2e35e208254673df9f0361e51 WatchSource:0}: Error finding container 34d5c5e5d1dd5734e52d73daf1236a97b1a0e3d2e35e208254673df9f0361e51: Status 404 returned error can't find the container with id 34d5c5e5d1dd5734e52d73daf1236a97b1a0e3d2e35e208254673df9f0361e51 Jul 02 19:03:57.095703 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:57.095613 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c2q69" event={"ID":"cf954a6b-7e4e-472a-8755-eafa3367d32c","Type":"ContainerStarted","Data":"4ec6c541d2894100c57a64c646fcc7c0cb96647435d6b6280abda1a290fae11d"} Jul 02 19:03:57.096894 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:57.096861 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xmmlt" event={"ID":"0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48","Type":"ContainerStarted","Data":"34d5c5e5d1dd5734e52d73daf1236a97b1a0e3d2e35e208254673df9f0361e51"} Jul 02 19:03:57.098086 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:57.098063 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t7n9j" event={"ID":"6f01d507-0625-477b-bf2f-7382a4c5bc9b","Type":"ContainerStarted","Data":"d812dea1544e3890e517a29e215c10cdbeab6600cf5adeb5ee3d9494a8e9c4b9"} Jul 02 19:03:57.743894 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:57.743807 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:59.114744 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:59.114701 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c2q69" event={"ID":"cf954a6b-7e4e-472a-8755-eafa3367d32c","Type":"ContainerStarted","Data":"aa15d10ba2c9ac08da65248a760509759dc1a14dc6accc91c9ed24aa105dc47f"} Jul 02 19:03:59.115169 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:59.114770 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c2q69" event={"ID":"cf954a6b-7e4e-472a-8755-eafa3367d32c","Type":"ContainerStarted","Data":"f55cc1e3c811bafac2afb95c4bbe890fdec72ce646238a6e66660bfdb69f1bfd"} Jul 02 19:03:59.133069 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:59.132482 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c2q69" podStartSLOduration=67.923438893 podStartE2EDuration="1m9.132462234s" totalImagesPullingTime="1.209023341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:02:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:03:59.131150049 +0000 UTC m=+68.928020241" watchObservedRunningTime="2026-07-02 19:03:59.132462234 +0000 UTC m=+68.929332428" Jul 02 19:03:59.217912 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:59.217875 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:03:59.322014 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:03:59.321977 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:01.095974 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:01.095949 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-69c68d759d-mmtpp" Jul 02 19:04:03.136372 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.136334 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xmmlt" event={"ID":"0b2d2be1-7e16-4d04-b95f-65c3ab6f9d48","Type":"ContainerStarted","Data":"18167ea02391ef03fa5ffddae45d7cebbefb1a9f69768ec626a19a40fb79247d"} Jul 02 19:04:03.136869 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.136825 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:04:03.138682 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.138659 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t7n9j" event={"ID":"6f01d507-0625-477b-bf2f-7382a4c5bc9b","Type":"ContainerStarted","Data":"3a22ee93f33a0ddd7cca389edff30ad539d649970eae250a1b4e4c7a2a8abacf"} Jul 02 19:04:03.143326 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.143302 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e118da-3b87-429d-9353-37ba79dc3205","Type":"ContainerStarted","Data":"86f275374a6fddeba90c29dcce8af798e3e237887bfd32c7658c62555b3853e6"} Jul 02 19:04:03.143436 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.143376 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e118da-3b87-429d-9353-37ba79dc3205","Type":"ContainerStarted","Data":"58f1f4339c793d93b0e735e9d44e8a27b90dab1ad909e534aa13edd8d0e8954b"} Jul 02 19:04:03.143436 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.143415 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e118da-3b87-429d-9353-37ba79dc3205","Type":"ContainerStarted","Data":"ff2cd220ad81bb8d2559316d93bf137dcd5a475d3f0c22c3283ec60d050b6021"} Jul 02 19:04:03.143551 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.143453 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e118da-3b87-429d-9353-37ba79dc3205","Type":"ContainerStarted","Data":"cc9969a998dc5f82e7041f83a48a0ca3aaec90e18aa03939233a8255fe1c56fa"} Jul 02 19:04:03.161174 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.161126 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xmmlt" podStartSLOduration=67.12413992 podStartE2EDuration="1m13.161112405s" totalImagesPullingTime="6.036972485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:02:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:04:03.160214795 +0000 UTC m=+72.957084987" watchObservedRunningTime="2026-07-02 19:04:03.161112405 +0000 UTC m=+72.957982596" Jul 02 19:04:03.180163 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.180124 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t7n9j" podStartSLOduration=66.096497298 podStartE2EDuration="1m12.180106749s" totalImagesPullingTime="6.083609451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:04:03.179422908 +0000 UTC m=+72.976293103" watchObservedRunningTime="2026-07-02 19:04:03.180106749 +0000 UTC m=+72.976976940" Jul 02 19:04:03.204724 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.204651 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:04:03.204960 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.204941 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:04:03.209688 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.209669 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:04:03.265823 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.265798 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:04:03.269610 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.269591 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:04:03.483017 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.482945 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:03.579697 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.579670 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:03.679990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.679961 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:03.737725 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:03.737648 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:04.149444 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:04.149408 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e118da-3b87-429d-9353-37ba79dc3205","Type":"ContainerStarted","Data":"15c11efafde965491f57d1108a3253258a2af6ebbcb2e90acd929355a6282602"} Jul 02 19:04:04.149794 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:04.149482 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d4e118da-3b87-429d-9353-37ba79dc3205","Type":"ContainerStarted","Data":"db8c46588adb027d09e6dfb96552b9e3d3bf3e0b2211f8a841a2131074978433"} Jul 02 19:04:04.153762 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:04.153741 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:04:04.182453 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:04.181201 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.275728419 podStartE2EDuration="10.181182313s" totalImagesPullingTime="7.88633244s" totalInitContainerRuntime="19.121454ms" isStatefulPod=false podCreationTimestamp="2026-07-02 19:03:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:04:04.179296742 +0000 UTC m=+73.976166984" watchObservedRunningTime="2026-07-02 19:04:04.181182313 +0000 UTC m=+73.978052508" Jul 02 19:04:04.235607 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:04.235580 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-576f7d85f-st92t"] Jul 02 19:04:04.354541 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:04.354496 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:04.500593 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:04.500562 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:04.579480 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:04.579451 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:04:04.604661 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:04.604631 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:05.560518 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:05.560481 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:05.657748 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:05.657711 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:06.775032 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:06.775006 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:06.878344 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:06.878314 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:14.595562 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:14.595449 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:14.694365 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:14.694330 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:17.888298 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:17.888250 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6dcb57cbbf-jkrpc" podUID="27802ccf-b4fe-4d73-a774-91fc256bd7ea" containerName="console" containerID="cri-o://7712051998bc753bcde6007df5b5dee53a5a8d2186fb6323dedcae21d05e7974" gracePeriod=15 Jul 02 19:04:18.132202 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.132178 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:04:18.205593 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.205509 2565 generic.go:356] "Generic (PLEG): container finished" podID="27802ccf-b4fe-4d73-a774-91fc256bd7ea" containerID="7712051998bc753bcde6007df5b5dee53a5a8d2186fb6323dedcae21d05e7974" exitCode=0 Jul 02 19:04:18.205593 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.205554 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcb57cbbf-jkrpc" Jul 02 19:04:18.205759 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.205556 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcb57cbbf-jkrpc" event={"ID":"27802ccf-b4fe-4d73-a774-91fc256bd7ea","Type":"ContainerDied","Data":"7712051998bc753bcde6007df5b5dee53a5a8d2186fb6323dedcae21d05e7974"} Jul 02 19:04:18.205759 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.205690 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcb57cbbf-jkrpc" event={"ID":"27802ccf-b4fe-4d73-a774-91fc256bd7ea","Type":"ContainerDied","Data":"1f76d59199c6a282eab396f8a2c101913af60c5b6499ddd65725826c5a19f8cf"} Jul 02 19:04:18.205759 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.205734 2565 scope.go:117] "RemoveContainer" containerID="7712051998bc753bcde6007df5b5dee53a5a8d2186fb6323dedcae21d05e7974" Jul 02 19:04:18.214357 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.214338 2565 scope.go:117] "RemoveContainer" containerID="7712051998bc753bcde6007df5b5dee53a5a8d2186fb6323dedcae21d05e7974" Jul 02 19:04:18.214611 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:04:18.214591 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7712051998bc753bcde6007df5b5dee53a5a8d2186fb6323dedcae21d05e7974\": container with ID starting with 7712051998bc753bcde6007df5b5dee53a5a8d2186fb6323dedcae21d05e7974 not found: ID does not exist" containerID="7712051998bc753bcde6007df5b5dee53a5a8d2186fb6323dedcae21d05e7974" Jul 02 19:04:18.214658 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.214624 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7712051998bc753bcde6007df5b5dee53a5a8d2186fb6323dedcae21d05e7974"} err="failed to get container status \"7712051998bc753bcde6007df5b5dee53a5a8d2186fb6323dedcae21d05e7974\": rpc error: code = NotFound desc = could not find container \"7712051998bc753bcde6007df5b5dee53a5a8d2186fb6323dedcae21d05e7974\": container with ID starting with 7712051998bc753bcde6007df5b5dee53a5a8d2186fb6323dedcae21d05e7974 not found: ID does not exist" Jul 02 19:04:18.312144 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.312115 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-serving-cert\") pod \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " Jul 02 19:04:18.312275 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.312161 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-service-ca\") pod \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " Jul 02 19:04:18.312275 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.312183 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zb99\" (UniqueName: \"kubernetes.io/projected/27802ccf-b4fe-4d73-a774-91fc256bd7ea-kube-api-access-4zb99\") pod \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " Jul 02 19:04:18.312275 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.312212 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-oauth-serving-cert\") pod \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " Jul 02 19:04:18.312275 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.312235 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-oauth-config\") pod \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " Jul 02 19:04:18.312275 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.312268 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-config\") pod \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\" (UID: \"27802ccf-b4fe-4d73-a774-91fc256bd7ea\") " Jul 02 19:04:18.312613 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.312577 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "27802ccf-b4fe-4d73-a774-91fc256bd7ea" (UID: "27802ccf-b4fe-4d73-a774-91fc256bd7ea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:04:18.312717 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.312621 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-service-ca" (OuterVolumeSpecName: "service-ca") pod "27802ccf-b4fe-4d73-a774-91fc256bd7ea" (UID: "27802ccf-b4fe-4d73-a774-91fc256bd7ea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:04:18.312717 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.312694 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-config" (OuterVolumeSpecName: "console-config") pod "27802ccf-b4fe-4d73-a774-91fc256bd7ea" (UID: "27802ccf-b4fe-4d73-a774-91fc256bd7ea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:04:18.314630 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.314607 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "27802ccf-b4fe-4d73-a774-91fc256bd7ea" (UID: "27802ccf-b4fe-4d73-a774-91fc256bd7ea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:04:18.314731 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.314673 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27802ccf-b4fe-4d73-a774-91fc256bd7ea-kube-api-access-4zb99" (OuterVolumeSpecName: "kube-api-access-4zb99") pod "27802ccf-b4fe-4d73-a774-91fc256bd7ea" (UID: "27802ccf-b4fe-4d73-a774-91fc256bd7ea"). InnerVolumeSpecName "kube-api-access-4zb99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:04:18.314731 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.314685 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "27802ccf-b4fe-4d73-a774-91fc256bd7ea" (UID: "27802ccf-b4fe-4d73-a774-91fc256bd7ea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:04:18.413463 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.413432 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-serving-cert\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:04:18.413463 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.413460 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:04:18.413463 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.413468 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zb99\" (UniqueName: \"kubernetes.io/projected/27802ccf-b4fe-4d73-a774-91fc256bd7ea-kube-api-access-4zb99\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:04:18.413671 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.413478 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-oauth-serving-cert\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:04:18.413671 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.413486 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-oauth-config\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:04:18.413671 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.413494 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27802ccf-b4fe-4d73-a774-91fc256bd7ea-console-config\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:04:18.525483 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.525458 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dcb57cbbf-jkrpc"] Jul 02 19:04:18.530856 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.530803 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dcb57cbbf-jkrpc"] Jul 02 19:04:18.761836 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:18.761800 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27802ccf-b4fe-4d73-a774-91fc256bd7ea" path="/var/lib/kubelet/pods/27802ccf-b4fe-4d73-a774-91fc256bd7ea/volumes" Jul 02 19:04:29.310486 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:29.310446 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:29.365329 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:29.365302 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:29.592017 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:29.591942 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:29.692623 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:29.692592 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:04:30.173816 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.173775 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-576f7d85f-st92t" podUID="d427d2f0-abca-4e47-b38b-c1f809edd977" containerName="console" containerID="cri-o://0babc0c924bdda9ec6b2a3125a97e6e6c7c10afdde16d8a78c519e8f801a965f" gracePeriod=15 Jul 02 19:04:30.407927 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.407903 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:04:30.506157 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.506130 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-trusted-ca-bundle\") pod \"d427d2f0-abca-4e47-b38b-c1f809edd977\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " Jul 02 19:04:30.506305 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.506166 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9zsc\" (UniqueName: \"kubernetes.io/projected/d427d2f0-abca-4e47-b38b-c1f809edd977-kube-api-access-s9zsc\") pod \"d427d2f0-abca-4e47-b38b-c1f809edd977\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " Jul 02 19:04:30.506305 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.506195 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d427d2f0-abca-4e47-b38b-c1f809edd977-console-oauth-config\") pod \"d427d2f0-abca-4e47-b38b-c1f809edd977\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " Jul 02 19:04:30.506305 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.506239 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-service-ca\") pod \"d427d2f0-abca-4e47-b38b-c1f809edd977\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " Jul 02 19:04:30.506305 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.506267 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d427d2f0-abca-4e47-b38b-c1f809edd977-console-serving-cert\") pod \"d427d2f0-abca-4e47-b38b-c1f809edd977\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " Jul 02 19:04:30.506305 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.506296 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-console-config\") pod \"d427d2f0-abca-4e47-b38b-c1f809edd977\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " Jul 02 19:04:30.506528 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.506329 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-oauth-serving-cert\") pod \"d427d2f0-abca-4e47-b38b-c1f809edd977\" (UID: \"d427d2f0-abca-4e47-b38b-c1f809edd977\") " Jul 02 19:04:30.506743 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.506711 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d427d2f0-abca-4e47-b38b-c1f809edd977" (UID: "d427d2f0-abca-4e47-b38b-c1f809edd977"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:04:30.506884 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.506762 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-service-ca" (OuterVolumeSpecName: "service-ca") pod "d427d2f0-abca-4e47-b38b-c1f809edd977" (UID: "d427d2f0-abca-4e47-b38b-c1f809edd977"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:04:30.506884 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.506778 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-console-config" (OuterVolumeSpecName: "console-config") pod "d427d2f0-abca-4e47-b38b-c1f809edd977" (UID: "d427d2f0-abca-4e47-b38b-c1f809edd977"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:04:30.506884 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.506810 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d427d2f0-abca-4e47-b38b-c1f809edd977" (UID: "d427d2f0-abca-4e47-b38b-c1f809edd977"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:04:30.508609 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.508587 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d427d2f0-abca-4e47-b38b-c1f809edd977-kube-api-access-s9zsc" (OuterVolumeSpecName: "kube-api-access-s9zsc") pod "d427d2f0-abca-4e47-b38b-c1f809edd977" (UID: "d427d2f0-abca-4e47-b38b-c1f809edd977"). InnerVolumeSpecName "kube-api-access-s9zsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:04:30.508681 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.508599 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d427d2f0-abca-4e47-b38b-c1f809edd977-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d427d2f0-abca-4e47-b38b-c1f809edd977" (UID: "d427d2f0-abca-4e47-b38b-c1f809edd977"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:04:30.508681 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.508647 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d427d2f0-abca-4e47-b38b-c1f809edd977-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d427d2f0-abca-4e47-b38b-c1f809edd977" (UID: "d427d2f0-abca-4e47-b38b-c1f809edd977"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:04:30.607174 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.607136 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:04:30.607174 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.607171 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d427d2f0-abca-4e47-b38b-c1f809edd977-console-serving-cert\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:04:30.607174 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.607180 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-console-config\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:04:30.607174 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.607189 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-oauth-serving-cert\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:04:30.607386 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.607198 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d427d2f0-abca-4e47-b38b-c1f809edd977-trusted-ca-bundle\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:04:30.607386 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.607207 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9zsc\" (UniqueName: \"kubernetes.io/projected/d427d2f0-abca-4e47-b38b-c1f809edd977-kube-api-access-s9zsc\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:04:30.607386 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:30.607215 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d427d2f0-abca-4e47-b38b-c1f809edd977-console-oauth-config\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:04:31.256350 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:31.256316 2565 generic.go:356] "Generic (PLEG): container finished" podID="d427d2f0-abca-4e47-b38b-c1f809edd977" containerID="0babc0c924bdda9ec6b2a3125a97e6e6c7c10afdde16d8a78c519e8f801a965f" exitCode=0 Jul 02 19:04:31.256536 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:31.256362 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-576f7d85f-st92t" Jul 02 19:04:31.256536 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:31.256403 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-576f7d85f-st92t" event={"ID":"d427d2f0-abca-4e47-b38b-c1f809edd977","Type":"ContainerDied","Data":"0babc0c924bdda9ec6b2a3125a97e6e6c7c10afdde16d8a78c519e8f801a965f"} Jul 02 19:04:31.256536 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:31.256467 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-576f7d85f-st92t" event={"ID":"d427d2f0-abca-4e47-b38b-c1f809edd977","Type":"ContainerDied","Data":"2b11e05cba19c2de5a11aaff38d1e94f578bc89a45ff6fb89678064e474d3c64"} Jul 02 19:04:31.256536 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:31.256495 2565 scope.go:117] "RemoveContainer" containerID="0babc0c924bdda9ec6b2a3125a97e6e6c7c10afdde16d8a78c519e8f801a965f" Jul 02 19:04:31.264916 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:31.264901 2565 scope.go:117] "RemoveContainer" containerID="0babc0c924bdda9ec6b2a3125a97e6e6c7c10afdde16d8a78c519e8f801a965f" Jul 02 19:04:31.265152 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:04:31.265135 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0babc0c924bdda9ec6b2a3125a97e6e6c7c10afdde16d8a78c519e8f801a965f\": container with ID starting with 0babc0c924bdda9ec6b2a3125a97e6e6c7c10afdde16d8a78c519e8f801a965f not found: ID does not exist" containerID="0babc0c924bdda9ec6b2a3125a97e6e6c7c10afdde16d8a78c519e8f801a965f" Jul 02 19:04:31.265193 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:31.265160 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0babc0c924bdda9ec6b2a3125a97e6e6c7c10afdde16d8a78c519e8f801a965f"} err="failed to get container status \"0babc0c924bdda9ec6b2a3125a97e6e6c7c10afdde16d8a78c519e8f801a965f\": rpc error: code = NotFound desc = could not find container \"0babc0c924bdda9ec6b2a3125a97e6e6c7c10afdde16d8a78c519e8f801a965f\": container with ID starting with 0babc0c924bdda9ec6b2a3125a97e6e6c7c10afdde16d8a78c519e8f801a965f not found: ID does not exist" Jul 02 19:04:31.273327 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:31.273307 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-576f7d85f-st92t"] Jul 02 19:04:31.275393 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:31.275376 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-576f7d85f-st92t"] Jul 02 19:04:32.765429 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:32.765397 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d427d2f0-abca-4e47-b38b-c1f809edd977" path="/var/lib/kubelet/pods/d427d2f0-abca-4e47-b38b-c1f809edd977/volumes" Jul 02 19:04:34.151828 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:34.151800 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xmmlt" Jul 02 19:04:54.579522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:54.579478 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:04:54.599156 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:54.599132 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:04:55.370821 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:04:55.370792 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 02 19:05:04.470514 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:04.470481 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:04.526304 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:04.526273 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:07.302417 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:07.302380 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:07.409864 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:07.409800 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:15.558104 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.558073 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67d54bc56d-77xsd"] Jul 02 19:05:15.558649 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.558553 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27802ccf-b4fe-4d73-a774-91fc256bd7ea" containerName="console" Jul 02 19:05:15.558649 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.558574 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="27802ccf-b4fe-4d73-a774-91fc256bd7ea" containerName="console" Jul 02 19:05:15.558649 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.558590 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d427d2f0-abca-4e47-b38b-c1f809edd977" containerName="console" Jul 02 19:05:15.558649 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.558598 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d427d2f0-abca-4e47-b38b-c1f809edd977" containerName="console" Jul 02 19:05:15.558879 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.558672 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="27802ccf-b4fe-4d73-a774-91fc256bd7ea" containerName="console" Jul 02 19:05:15.558879 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.558686 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d427d2f0-abca-4e47-b38b-c1f809edd977" containerName="console" Jul 02 19:05:15.560583 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.560562 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.573333 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.573308 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d54bc56d-77xsd"] Jul 02 19:05:15.582242 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.582213 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-service-ca\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.582386 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.582259 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-trusted-ca-bundle\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.582386 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.582320 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-oauth-serving-cert\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.582386 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.582347 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxf8p\" (UniqueName: \"kubernetes.io/projected/4a882d8b-9f99-45f4-8158-e043b9885674-kube-api-access-fxf8p\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.582386 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.582371 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a882d8b-9f99-45f4-8158-e043b9885674-console-oauth-config\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.582563 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.582398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a882d8b-9f99-45f4-8158-e043b9885674-console-serving-cert\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.582563 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.582415 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-console-config\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.682700 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.682660 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:15.682897 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.682800 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a882d8b-9f99-45f4-8158-e043b9885674-console-serving-cert\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.682897 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.682824 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-console-config\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.683007 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.682918 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-service-ca\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.683007 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.682938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-trusted-ca-bundle\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.683099 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.683006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-oauth-serving-cert\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.683099 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.683046 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxf8p\" (UniqueName: \"kubernetes.io/projected/4a882d8b-9f99-45f4-8158-e043b9885674-kube-api-access-fxf8p\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.683099 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.683084 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a882d8b-9f99-45f4-8158-e043b9885674-console-oauth-config\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.683820 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.683791 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-oauth-serving-cert\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.683999 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.683894 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-trusted-ca-bundle\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.684062 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.683927 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-service-ca\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.684062 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.683928 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-console-config\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.685627 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.685600 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a882d8b-9f99-45f4-8158-e043b9885674-console-serving-cert\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.685879 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.685861 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a882d8b-9f99-45f4-8158-e043b9885674-console-oauth-config\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.692760 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.692736 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxf8p\" (UniqueName: \"kubernetes.io/projected/4a882d8b-9f99-45f4-8158-e043b9885674-kube-api-access-fxf8p\") pod \"console-67d54bc56d-77xsd\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:15.782449 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.782405 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:15.869944 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:15.869835 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:16.007556 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:16.007524 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67d54bc56d-77xsd"] Jul 02 19:05:16.010397 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:05:16.010369 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a882d8b_9f99_45f4_8158_e043b9885674.slice/crio-b7e07a03613beb24913662921bac340eedcd65e3a9b3aa305a7d0b79c45d9d2a WatchSource:0}: Error finding container b7e07a03613beb24913662921bac340eedcd65e3a9b3aa305a7d0b79c45d9d2a: Status 404 returned error can't find the container with id b7e07a03613beb24913662921bac340eedcd65e3a9b3aa305a7d0b79c45d9d2a Jul 02 19:05:16.433755 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:16.433722 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d54bc56d-77xsd" event={"ID":"4a882d8b-9f99-45f4-8158-e043b9885674","Type":"ContainerStarted","Data":"48e561ff96b5c74a2fc2f96528378dd175439565b5c6fca783934d16b32ef387"} Jul 02 19:05:16.433938 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:16.433776 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d54bc56d-77xsd" event={"ID":"4a882d8b-9f99-45f4-8158-e043b9885674","Type":"ContainerStarted","Data":"b7e07a03613beb24913662921bac340eedcd65e3a9b3aa305a7d0b79c45d9d2a"} Jul 02 19:05:16.454953 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:16.454900 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67d54bc56d-77xsd" podStartSLOduration=1.45488267 podStartE2EDuration="1.45488267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:05:16.453785195 +0000 UTC m=+146.250655386" watchObservedRunningTime="2026-07-02 19:05:16.45488267 +0000 UTC m=+146.251752860" Jul 02 19:05:19.923835 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:19.923798 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:22.586128 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:22.586091 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:22.643500 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:22.643473 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:25.870186 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:25.870147 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:25.870563 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:25.870211 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:25.875078 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:25.875056 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:26.473249 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.473218 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:05:26.521603 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.521571 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8598fddfc9-cm9bn"] Jul 02 19:05:26.643946 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.643913 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cc866d7f9-kd9cx"] Jul 02 19:05:26.647061 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.647046 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.652190 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.652165 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:26.659153 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.659134 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cc866d7f9-kd9cx"] Jul 02 19:05:26.663970 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.663950 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-service-ca\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.664046 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.663989 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-oauth-serving-cert\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.664084 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.664044 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-serving-cert\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.664118 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.664087 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-oauth-config\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.664151 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.664128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc8n5\" (UniqueName: \"kubernetes.io/projected/6aad3414-2ec6-4b06-8a66-604aa95a55ca-kube-api-access-jc8n5\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.664186 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.664153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-trusted-ca-bundle\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.664186 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.664172 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-config\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.764495 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.764465 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-service-ca\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.764661 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.764505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-oauth-serving-cert\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.764661 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.764532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-serving-cert\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.764661 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.764568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-oauth-config\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.764661 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.764606 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc8n5\" (UniqueName: \"kubernetes.io/projected/6aad3414-2ec6-4b06-8a66-604aa95a55ca-kube-api-access-jc8n5\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.764901 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.764824 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-trusted-ca-bundle\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.764961 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.764946 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-config\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.765345 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.765286 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-oauth-serving-cert\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.765634 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.765612 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-config\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.765947 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.765915 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-service-ca\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.766032 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.765959 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-trusted-ca-bundle\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.767190 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.767165 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-oauth-config\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.767479 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.767460 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-serving-cert\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.776165 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.776146 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc8n5\" (UniqueName: \"kubernetes.io/projected/6aad3414-2ec6-4b06-8a66-604aa95a55ca-kube-api-access-jc8n5\") pod \"console-5cc866d7f9-kd9cx\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.956661 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.956615 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:26.978120 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:26.978096 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:27.087303 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:27.087279 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cc866d7f9-kd9cx"] Jul 02 19:05:27.089253 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:05:27.089223 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aad3414_2ec6_4b06_8a66_604aa95a55ca.slice/crio-9c5de005411648c8ff0279cc0744333d0b27b5475eee1fcb7879e45a274edcad WatchSource:0}: Error finding container 9c5de005411648c8ff0279cc0744333d0b27b5475eee1fcb7879e45a274edcad: Status 404 returned error can't find the container with id 9c5de005411648c8ff0279cc0744333d0b27b5475eee1fcb7879e45a274edcad Jul 02 19:05:27.474353 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:27.474262 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc866d7f9-kd9cx" event={"ID":"6aad3414-2ec6-4b06-8a66-604aa95a55ca","Type":"ContainerStarted","Data":"2e77d2f9fa076fac1478bbe0225be67b6dd41d2a37e81ff74b7d9885ea6654b2"} Jul 02 19:05:27.474353 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:27.474317 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc866d7f9-kd9cx" event={"ID":"6aad3414-2ec6-4b06-8a66-604aa95a55ca","Type":"ContainerStarted","Data":"9c5de005411648c8ff0279cc0744333d0b27b5475eee1fcb7879e45a274edcad"} Jul 02 19:05:27.493207 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:27.493165 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5cc866d7f9-kd9cx" podStartSLOduration=1.493150038 podStartE2EDuration="1.493150038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:05:27.491760388 +0000 UTC m=+157.288630580" watchObservedRunningTime="2026-07-02 19:05:27.493150038 +0000 UTC m=+157.290020228" Jul 02 19:05:34.590131 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:34.590095 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:34.645138 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:34.645105 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:36.957253 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:36.957206 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:36.957643 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:36.957306 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:36.961896 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:36.961876 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:37.514003 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:37.513972 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:05:37.570166 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:37.570134 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67d54bc56d-77xsd"] Jul 02 19:05:37.674041 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:37.674013 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:37.726544 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:37.726514 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:37.782289 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:37.782209 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:44.606209 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:44.606120 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:44.706527 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:44.706484 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:51.542305 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.542262 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-8598fddfc9-cm9bn" podUID="580ec438-54e1-4767-8f05-55cec36ea8fa" containerName="console" containerID="cri-o://f787168787b1fbb50fb9b4ad1551f9e841018f52bc0914c0567872591f782616" gracePeriod=15 Jul 02 19:05:51.784044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.784018 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:05:51.870245 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.870181 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-oauth-serving-cert\") pod \"580ec438-54e1-4767-8f05-55cec36ea8fa\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " Jul 02 19:05:51.870387 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.870273 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/580ec438-54e1-4767-8f05-55cec36ea8fa-console-oauth-config\") pod \"580ec438-54e1-4767-8f05-55cec36ea8fa\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " Jul 02 19:05:51.870387 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.870308 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg926\" (UniqueName: \"kubernetes.io/projected/580ec438-54e1-4767-8f05-55cec36ea8fa-kube-api-access-hg926\") pod \"580ec438-54e1-4767-8f05-55cec36ea8fa\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " Jul 02 19:05:51.870387 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.870335 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-trusted-ca-bundle\") pod \"580ec438-54e1-4767-8f05-55cec36ea8fa\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " Jul 02 19:05:51.870387 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.870360 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-console-config\") pod \"580ec438-54e1-4767-8f05-55cec36ea8fa\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " Jul 02 19:05:51.870588 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.870387 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-service-ca\") pod \"580ec438-54e1-4767-8f05-55cec36ea8fa\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " Jul 02 19:05:51.870588 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.870413 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/580ec438-54e1-4767-8f05-55cec36ea8fa-console-serving-cert\") pod \"580ec438-54e1-4767-8f05-55cec36ea8fa\" (UID: \"580ec438-54e1-4767-8f05-55cec36ea8fa\") " Jul 02 19:05:51.870689 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.870566 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "580ec438-54e1-4767-8f05-55cec36ea8fa" (UID: "580ec438-54e1-4767-8f05-55cec36ea8fa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:05:51.870781 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.870755 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-service-ca" (OuterVolumeSpecName: "service-ca") pod "580ec438-54e1-4767-8f05-55cec36ea8fa" (UID: "580ec438-54e1-4767-8f05-55cec36ea8fa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:05:51.870781 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.870764 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "580ec438-54e1-4767-8f05-55cec36ea8fa" (UID: "580ec438-54e1-4767-8f05-55cec36ea8fa"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:05:51.871332 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.871225 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-console-config" (OuterVolumeSpecName: "console-config") pod "580ec438-54e1-4767-8f05-55cec36ea8fa" (UID: "580ec438-54e1-4767-8f05-55cec36ea8fa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:05:51.872714 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.872690 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/580ec438-54e1-4767-8f05-55cec36ea8fa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "580ec438-54e1-4767-8f05-55cec36ea8fa" (UID: "580ec438-54e1-4767-8f05-55cec36ea8fa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:05:51.872816 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.872741 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580ec438-54e1-4767-8f05-55cec36ea8fa-kube-api-access-hg926" (OuterVolumeSpecName: "kube-api-access-hg926") pod "580ec438-54e1-4767-8f05-55cec36ea8fa" (UID: "580ec438-54e1-4767-8f05-55cec36ea8fa"). InnerVolumeSpecName "kube-api-access-hg926". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:05:51.872816 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.872760 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/580ec438-54e1-4767-8f05-55cec36ea8fa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "580ec438-54e1-4767-8f05-55cec36ea8fa" (UID: "580ec438-54e1-4767-8f05-55cec36ea8fa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:05:51.972048 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.972018 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/580ec438-54e1-4767-8f05-55cec36ea8fa-console-oauth-config\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:05:51.972048 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.972049 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg926\" (UniqueName: \"kubernetes.io/projected/580ec438-54e1-4767-8f05-55cec36ea8fa-kube-api-access-hg926\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:05:51.972191 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.972063 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-trusted-ca-bundle\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:05:51.972191 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.972075 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-console-config\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:05:51.972191 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.972087 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:05:51.972191 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.972097 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/580ec438-54e1-4767-8f05-55cec36ea8fa-console-serving-cert\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:05:51.972191 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:51.972109 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/580ec438-54e1-4767-8f05-55cec36ea8fa-oauth-serving-cert\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:05:52.566748 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:52.566712 2565 generic.go:356] "Generic (PLEG): container finished" podID="580ec438-54e1-4767-8f05-55cec36ea8fa" containerID="f787168787b1fbb50fb9b4ad1551f9e841018f52bc0914c0567872591f782616" exitCode=0 Jul 02 19:05:52.567183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:52.566758 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8598fddfc9-cm9bn" Jul 02 19:05:52.567183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:52.566799 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8598fddfc9-cm9bn" event={"ID":"580ec438-54e1-4767-8f05-55cec36ea8fa","Type":"ContainerDied","Data":"f787168787b1fbb50fb9b4ad1551f9e841018f52bc0914c0567872591f782616"} Jul 02 19:05:52.567183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:52.566874 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8598fddfc9-cm9bn" event={"ID":"580ec438-54e1-4767-8f05-55cec36ea8fa","Type":"ContainerDied","Data":"8254f377f55a4c07442d8535fc15c07591a7eccfbb0c4f1723d560c9809c04be"} Jul 02 19:05:52.567183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:52.566896 2565 scope.go:117] "RemoveContainer" containerID="f787168787b1fbb50fb9b4ad1551f9e841018f52bc0914c0567872591f782616" Jul 02 19:05:52.575542 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:52.575528 2565 scope.go:117] "RemoveContainer" containerID="f787168787b1fbb50fb9b4ad1551f9e841018f52bc0914c0567872591f782616" Jul 02 19:05:52.575809 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:05:52.575789 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f787168787b1fbb50fb9b4ad1551f9e841018f52bc0914c0567872591f782616\": container with ID starting with f787168787b1fbb50fb9b4ad1551f9e841018f52bc0914c0567872591f782616 not found: ID does not exist" containerID="f787168787b1fbb50fb9b4ad1551f9e841018f52bc0914c0567872591f782616" Jul 02 19:05:52.575893 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:52.575822 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f787168787b1fbb50fb9b4ad1551f9e841018f52bc0914c0567872591f782616"} err="failed to get container status \"f787168787b1fbb50fb9b4ad1551f9e841018f52bc0914c0567872591f782616\": rpc error: code = NotFound desc = could not find container \"f787168787b1fbb50fb9b4ad1551f9e841018f52bc0914c0567872591f782616\": container with ID starting with f787168787b1fbb50fb9b4ad1551f9e841018f52bc0914c0567872591f782616 not found: ID does not exist" Jul 02 19:05:52.587225 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:52.587201 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8598fddfc9-cm9bn"] Jul 02 19:05:52.589835 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:52.589817 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8598fddfc9-cm9bn"] Jul 02 19:05:52.764332 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:52.763774 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="580ec438-54e1-4767-8f05-55cec36ea8fa" path="/var/lib/kubelet/pods/580ec438-54e1-4767-8f05-55cec36ea8fa/volumes" Jul 02 19:05:59.854019 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:59.853985 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:05:59.976488 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:05:59.976459 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:06:02.591012 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.590965 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-67d54bc56d-77xsd" podUID="4a882d8b-9f99-45f4-8158-e043b9885674" containerName="console" containerID="cri-o://48e561ff96b5c74a2fc2f96528378dd175439565b5c6fca783934d16b32ef387" gracePeriod=15 Jul 02 19:06:02.828677 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.828657 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:06:02.964670 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.964568 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxf8p\" (UniqueName: \"kubernetes.io/projected/4a882d8b-9f99-45f4-8158-e043b9885674-kube-api-access-fxf8p\") pod \"4a882d8b-9f99-45f4-8158-e043b9885674\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " Jul 02 19:06:02.964670 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.964628 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-oauth-serving-cert\") pod \"4a882d8b-9f99-45f4-8158-e043b9885674\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " Jul 02 19:06:02.964670 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.964646 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a882d8b-9f99-45f4-8158-e043b9885674-console-oauth-config\") pod \"4a882d8b-9f99-45f4-8158-e043b9885674\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " Jul 02 19:06:02.964670 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.964671 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-service-ca\") pod \"4a882d8b-9f99-45f4-8158-e043b9885674\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " Jul 02 19:06:02.965032 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.964730 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-console-config\") pod \"4a882d8b-9f99-45f4-8158-e043b9885674\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " Jul 02 19:06:02.965032 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.964765 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-trusted-ca-bundle\") pod \"4a882d8b-9f99-45f4-8158-e043b9885674\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " Jul 02 19:06:02.965032 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.964791 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a882d8b-9f99-45f4-8158-e043b9885674-console-serving-cert\") pod \"4a882d8b-9f99-45f4-8158-e043b9885674\" (UID: \"4a882d8b-9f99-45f4-8158-e043b9885674\") " Jul 02 19:06:02.965177 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.965028 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4a882d8b-9f99-45f4-8158-e043b9885674" (UID: "4a882d8b-9f99-45f4-8158-e043b9885674"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:06:02.965234 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.965198 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4a882d8b-9f99-45f4-8158-e043b9885674" (UID: "4a882d8b-9f99-45f4-8158-e043b9885674"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:06:02.965333 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.965299 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-service-ca" (OuterVolumeSpecName: "service-ca") pod "4a882d8b-9f99-45f4-8158-e043b9885674" (UID: "4a882d8b-9f99-45f4-8158-e043b9885674"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:06:02.965451 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.965373 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-console-config" (OuterVolumeSpecName: "console-config") pod "4a882d8b-9f99-45f4-8158-e043b9885674" (UID: "4a882d8b-9f99-45f4-8158-e043b9885674"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:06:02.967262 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.967245 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a882d8b-9f99-45f4-8158-e043b9885674-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4a882d8b-9f99-45f4-8158-e043b9885674" (UID: "4a882d8b-9f99-45f4-8158-e043b9885674"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:06:02.967512 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.967485 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a882d8b-9f99-45f4-8158-e043b9885674-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4a882d8b-9f99-45f4-8158-e043b9885674" (UID: "4a882d8b-9f99-45f4-8158-e043b9885674"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:06:02.967512 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:02.967501 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a882d8b-9f99-45f4-8158-e043b9885674-kube-api-access-fxf8p" (OuterVolumeSpecName: "kube-api-access-fxf8p") pod "4a882d8b-9f99-45f4-8158-e043b9885674" (UID: "4a882d8b-9f99-45f4-8158-e043b9885674"). InnerVolumeSpecName "kube-api-access-fxf8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:06:03.065933 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.065890 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-trusted-ca-bundle\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:06:03.065933 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.065928 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a882d8b-9f99-45f4-8158-e043b9885674-console-serving-cert\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:06:03.065933 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.065937 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxf8p\" (UniqueName: \"kubernetes.io/projected/4a882d8b-9f99-45f4-8158-e043b9885674-kube-api-access-fxf8p\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:06:03.065933 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.065947 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-oauth-serving-cert\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:06:03.066171 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.065955 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a882d8b-9f99-45f4-8158-e043b9885674-console-oauth-config\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:06:03.066171 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.065964 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:06:03.066171 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.065972 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a882d8b-9f99-45f4-8158-e043b9885674-console-config\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:06:03.608223 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.608190 2565 generic.go:356] "Generic (PLEG): container finished" podID="4a882d8b-9f99-45f4-8158-e043b9885674" containerID="48e561ff96b5c74a2fc2f96528378dd175439565b5c6fca783934d16b32ef387" exitCode=0 Jul 02 19:06:03.608627 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.608244 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67d54bc56d-77xsd" Jul 02 19:06:03.608627 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.608267 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d54bc56d-77xsd" event={"ID":"4a882d8b-9f99-45f4-8158-e043b9885674","Type":"ContainerDied","Data":"48e561ff96b5c74a2fc2f96528378dd175439565b5c6fca783934d16b32ef387"} Jul 02 19:06:03.608627 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.608307 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67d54bc56d-77xsd" event={"ID":"4a882d8b-9f99-45f4-8158-e043b9885674","Type":"ContainerDied","Data":"b7e07a03613beb24913662921bac340eedcd65e3a9b3aa305a7d0b79c45d9d2a"} Jul 02 19:06:03.608627 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.608326 2565 scope.go:117] "RemoveContainer" containerID="48e561ff96b5c74a2fc2f96528378dd175439565b5c6fca783934d16b32ef387" Jul 02 19:06:03.616699 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.616684 2565 scope.go:117] "RemoveContainer" containerID="48e561ff96b5c74a2fc2f96528378dd175439565b5c6fca783934d16b32ef387" Jul 02 19:06:03.616972 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:06:03.616952 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e561ff96b5c74a2fc2f96528378dd175439565b5c6fca783934d16b32ef387\": container with ID starting with 48e561ff96b5c74a2fc2f96528378dd175439565b5c6fca783934d16b32ef387 not found: ID does not exist" containerID="48e561ff96b5c74a2fc2f96528378dd175439565b5c6fca783934d16b32ef387" Jul 02 19:06:03.617035 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.616978 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e561ff96b5c74a2fc2f96528378dd175439565b5c6fca783934d16b32ef387"} err="failed to get container status \"48e561ff96b5c74a2fc2f96528378dd175439565b5c6fca783934d16b32ef387\": rpc error: code = NotFound desc = could not find container \"48e561ff96b5c74a2fc2f96528378dd175439565b5c6fca783934d16b32ef387\": container with ID starting with 48e561ff96b5c74a2fc2f96528378dd175439565b5c6fca783934d16b32ef387 not found: ID does not exist" Jul 02 19:06:03.629400 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.629376 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67d54bc56d-77xsd"] Jul 02 19:06:03.635560 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:03.635540 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67d54bc56d-77xsd"] Jul 02 19:06:04.761602 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:04.761572 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a882d8b-9f99-45f4-8158-e043b9885674" path="/var/lib/kubelet/pods/4a882d8b-9f99-45f4-8158-e043b9885674/volumes" Jul 02 19:06:18.293883 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:18.293834 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:06:18.344380 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:18.344354 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:06:19.282532 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:19.282487 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:06:19.409181 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:19.409147 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:06:46.552933 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:46.552896 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:06:47.925891 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:47.925826 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:06:50.147519 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:50.147485 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:06:50.249308 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:50.249273 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:06:55.153000 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:55.152965 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:06:55.213079 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:55.213048 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:06:58.337980 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:58.337944 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:06:58.429004 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:58.428978 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:06:58.767526 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:58.767491 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:06:58.923903 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:58.923869 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:06:59.333786 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:06:59.333752 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:00.332385 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:00.332347 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:00.733801 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:00.733728 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:06.432524 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:06.432490 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:06.486179 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:06.486149 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:09.146486 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:09.146447 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:22.037468 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:22.037382 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:22.132348 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:22.132313 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:24.258394 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.258362 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qxxzl"] Jul 02 19:07:24.258748 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.258624 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="580ec438-54e1-4767-8f05-55cec36ea8fa" containerName="console" Jul 02 19:07:24.258748 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.258634 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="580ec438-54e1-4767-8f05-55cec36ea8fa" containerName="console" Jul 02 19:07:24.258748 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.258644 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a882d8b-9f99-45f4-8158-e043b9885674" containerName="console" Jul 02 19:07:24.258748 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.258650 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a882d8b-9f99-45f4-8158-e043b9885674" containerName="console" Jul 02 19:07:24.258748 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.258688 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="580ec438-54e1-4767-8f05-55cec36ea8fa" containerName="console" Jul 02 19:07:24.258748 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.258697 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a882d8b-9f99-45f4-8158-e043b9885674" containerName="console" Jul 02 19:07:24.260431 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.260416 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qxxzl" Jul 02 19:07:24.263612 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.263587 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 02 19:07:24.263612 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.263602 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-tf4qn\"" Jul 02 19:07:24.264497 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.264480 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:07:24.271974 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.271954 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qxxzl"] Jul 02 19:07:24.361489 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.361465 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb88d\" (UniqueName: \"kubernetes.io/projected/5f912e6c-1f59-482a-a0cb-d91a4ff1a2c6-kube-api-access-wb88d\") pod \"openshift-lws-operator-fd8ccff4c-qxxzl\" (UID: \"5f912e6c-1f59-482a-a0cb-d91a4ff1a2c6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qxxzl" Jul 02 19:07:24.361579 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.361521 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f912e6c-1f59-482a-a0cb-d91a4ff1a2c6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qxxzl\" (UID: \"5f912e6c-1f59-482a-a0cb-d91a4ff1a2c6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qxxzl" Jul 02 19:07:24.462763 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.462739 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f912e6c-1f59-482a-a0cb-d91a4ff1a2c6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qxxzl\" (UID: \"5f912e6c-1f59-482a-a0cb-d91a4ff1a2c6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qxxzl" Jul 02 19:07:24.462891 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.462780 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb88d\" (UniqueName: \"kubernetes.io/projected/5f912e6c-1f59-482a-a0cb-d91a4ff1a2c6-kube-api-access-wb88d\") pod \"openshift-lws-operator-fd8ccff4c-qxxzl\" (UID: \"5f912e6c-1f59-482a-a0cb-d91a4ff1a2c6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qxxzl" Jul 02 19:07:24.463173 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.463156 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f912e6c-1f59-482a-a0cb-d91a4ff1a2c6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-qxxzl\" (UID: \"5f912e6c-1f59-482a-a0cb-d91a4ff1a2c6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qxxzl" Jul 02 19:07:24.470554 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.470531 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb88d\" (UniqueName: \"kubernetes.io/projected/5f912e6c-1f59-482a-a0cb-d91a4ff1a2c6-kube-api-access-wb88d\") pod \"openshift-lws-operator-fd8ccff4c-qxxzl\" (UID: \"5f912e6c-1f59-482a-a0cb-d91a4ff1a2c6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qxxzl" Jul 02 19:07:24.581470 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.581412 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qxxzl" Jul 02 19:07:24.713118 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.713091 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qxxzl"] Jul 02 19:07:24.717220 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:07:24.717155 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f912e6c_1f59_482a_a0cb_d91a4ff1a2c6.slice/crio-1c4437273cf7b016eef4ed1c37b4c36700a98c183ac3bfa08016d247eda69dea WatchSource:0}: Error finding container 1c4437273cf7b016eef4ed1c37b4c36700a98c183ac3bfa08016d247eda69dea: Status 404 returned error can't find the container with id 1c4437273cf7b016eef4ed1c37b4c36700a98c183ac3bfa08016d247eda69dea Jul 02 19:07:24.886880 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:24.886773 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qxxzl" event={"ID":"5f912e6c-1f59-482a-a0cb-d91a4ff1a2c6","Type":"ContainerStarted","Data":"1c4437273cf7b016eef4ed1c37b4c36700a98c183ac3bfa08016d247eda69dea"} Jul 02 19:07:27.902061 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:27.902028 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qxxzl" event={"ID":"5f912e6c-1f59-482a-a0cb-d91a4ff1a2c6","Type":"ContainerStarted","Data":"a22502bdc9f6ce23ad1a791978d17edbee2d12c752718604ac518afea34f5bf2"} Jul 02 19:07:27.919761 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:27.919664 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-qxxzl" podStartSLOduration=0.97202884 podStartE2EDuration="3.919650182s" totalImagesPullingTime="2.947621342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:07:27.919476602 +0000 UTC m=+277.716346794" watchObservedRunningTime="2026-07-02 19:07:27.919650182 +0000 UTC m=+277.716520373" Jul 02 19:07:28.649394 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:28.649360 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:28.752094 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:28.752063 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:31.032711 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:31.032665 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:39.851221 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:39.851192 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:46.372726 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:46.372676 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:46.473102 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:46.473066 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:46.968774 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:46.968743 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:47.290342 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.290308 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:47.391364 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.391322 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:47.450178 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.450138 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf"] Jul 02 19:07:47.455677 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.455651 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.459074 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.459052 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 02 19:07:47.459334 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.459317 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 02 19:07:47.459443 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.459341 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hgcj5\"" Jul 02 19:07:47.459443 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.459317 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 02 19:07:47.463176 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.462956 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 02 19:07:47.492541 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.492515 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf"] Jul 02 19:07:47.516886 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.516865 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrwd4\" (UniqueName: \"kubernetes.io/projected/cf3f821b-654f-497b-ae98-9760fb90c79b-kube-api-access-hrwd4\") pod \"opendatahub-operator-controller-manager-6cd5847cc4-t4khf\" (UID: \"cf3f821b-654f-497b-ae98-9760fb90c79b\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.516999 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.516909 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf3f821b-654f-497b-ae98-9760fb90c79b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cd5847cc4-t4khf\" (UID: \"cf3f821b-654f-497b-ae98-9760fb90c79b\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.516999 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.516928 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf3f821b-654f-497b-ae98-9760fb90c79b-tmp\") pod \"opendatahub-operator-controller-manager-6cd5847cc4-t4khf\" (UID: \"cf3f821b-654f-497b-ae98-9760fb90c79b\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.516999 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.516945 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf3f821b-654f-497b-ae98-9760fb90c79b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cd5847cc4-t4khf\" (UID: \"cf3f821b-654f-497b-ae98-9760fb90c79b\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.517138 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.517033 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf3f821b-654f-497b-ae98-9760fb90c79b-manifests\") pod \"opendatahub-operator-controller-manager-6cd5847cc4-t4khf\" (UID: \"cf3f821b-654f-497b-ae98-9760fb90c79b\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.617764 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.617673 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf3f821b-654f-497b-ae98-9760fb90c79b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cd5847cc4-t4khf\" (UID: \"cf3f821b-654f-497b-ae98-9760fb90c79b\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.617764 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.617708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf3f821b-654f-497b-ae98-9760fb90c79b-tmp\") pod \"opendatahub-operator-controller-manager-6cd5847cc4-t4khf\" (UID: \"cf3f821b-654f-497b-ae98-9760fb90c79b\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.617764 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.617734 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf3f821b-654f-497b-ae98-9760fb90c79b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cd5847cc4-t4khf\" (UID: \"cf3f821b-654f-497b-ae98-9760fb90c79b\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.618076 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.617778 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf3f821b-654f-497b-ae98-9760fb90c79b-manifests\") pod \"opendatahub-operator-controller-manager-6cd5847cc4-t4khf\" (UID: \"cf3f821b-654f-497b-ae98-9760fb90c79b\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.618076 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.617805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrwd4\" (UniqueName: \"kubernetes.io/projected/cf3f821b-654f-497b-ae98-9760fb90c79b-kube-api-access-hrwd4\") pod \"opendatahub-operator-controller-manager-6cd5847cc4-t4khf\" (UID: \"cf3f821b-654f-497b-ae98-9760fb90c79b\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.618243 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.618222 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf3f821b-654f-497b-ae98-9760fb90c79b-manifests\") pod \"opendatahub-operator-controller-manager-6cd5847cc4-t4khf\" (UID: \"cf3f821b-654f-497b-ae98-9760fb90c79b\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.620147 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.620126 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf3f821b-654f-497b-ae98-9760fb90c79b-tmp\") pod \"opendatahub-operator-controller-manager-6cd5847cc4-t4khf\" (UID: \"cf3f821b-654f-497b-ae98-9760fb90c79b\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.620429 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.620410 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf3f821b-654f-497b-ae98-9760fb90c79b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cd5847cc4-t4khf\" (UID: \"cf3f821b-654f-497b-ae98-9760fb90c79b\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.620489 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.620461 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf3f821b-654f-497b-ae98-9760fb90c79b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cd5847cc4-t4khf\" (UID: \"cf3f821b-654f-497b-ae98-9760fb90c79b\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.625797 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.625778 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:47.628452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.628424 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrwd4\" (UniqueName: \"kubernetes.io/projected/cf3f821b-654f-497b-ae98-9760fb90c79b-kube-api-access-hrwd4\") pod \"opendatahub-operator-controller-manager-6cd5847cc4-t4khf\" (UID: \"cf3f821b-654f-497b-ae98-9760fb90c79b\") " pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.722479 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.722428 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:47.766788 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.766763 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:47.824332 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.824307 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:47.914898 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.914864 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf"] Jul 02 19:07:47.917354 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:07:47.917325 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3f821b_654f_497b_ae98_9760fb90c79b.slice/crio-220b6df123af1c5b934d45c7071c66c7dc33df71da899b9811e70e27b7db0c0b WatchSource:0}: Error finding container 220b6df123af1c5b934d45c7071c66c7dc33df71da899b9811e70e27b7db0c0b: Status 404 returned error can't find the container with id 220b6df123af1c5b934d45c7071c66c7dc33df71da899b9811e70e27b7db0c0b Jul 02 19:07:47.920798 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.920775 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:47.975784 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:47.975757 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" event={"ID":"cf3f821b-654f-497b-ae98-9760fb90c79b","Type":"ContainerStarted","Data":"220b6df123af1c5b934d45c7071c66c7dc33df71da899b9811e70e27b7db0c0b"} Jul 02 19:07:48.059332 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:48.059268 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:48.163932 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:48.163898 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:49.396692 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:49.396657 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:49.498582 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:49.498533 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:07:51.109247 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:51.109212 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:07:51.109247 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:51.109243 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:07:51.116578 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:51.116557 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 02 19:07:51.994358 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:51.994324 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" event={"ID":"cf3f821b-654f-497b-ae98-9760fb90c79b","Type":"ContainerStarted","Data":"45d07b8a9e1ded213176535a35342412dc9c4bc933f30d17b6e489f97c464939"} Jul 02 19:07:53.002830 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:53.002790 2565 generic.go:356] "Generic (PLEG): container finished" podID="cf3f821b-654f-497b-ae98-9760fb90c79b" containerID="45d07b8a9e1ded213176535a35342412dc9c4bc933f30d17b6e489f97c464939" exitCode=0 Jul 02 19:07:53.003320 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:53.002888 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" event={"ID":"cf3f821b-654f-497b-ae98-9760fb90c79b","Type":"ContainerDied","Data":"45d07b8a9e1ded213176535a35342412dc9c4bc933f30d17b6e489f97c464939"} Jul 02 19:07:53.003467 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:53.003449 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:07:54.008568 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:54.008533 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" event={"ID":"cf3f821b-654f-497b-ae98-9760fb90c79b","Type":"ContainerStarted","Data":"b863bc031f42e5f7e7a5c9c61123953e0da5db6d77c4f98010ec5287f30c9a23"} Jul 02 19:07:54.008954 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:54.008668 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:07:54.032968 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:07:54.032922 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" podStartSLOduration=2.392021543 podStartE2EDuration="7.032908966s" totalImagesPullingTime="3.555434717s" totalInitContainerRuntime="1.085452706s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:07:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:07:54.031041518 +0000 UTC m=+303.827911735" watchObservedRunningTime="2026-07-02 19:07:54.032908966 +0000 UTC m=+303.829779157" Jul 02 19:08:05.014637 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:05.014608 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cd5847cc4-t4khf" Jul 02 19:08:06.877366 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:06.877309 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:07.118901 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:07.118870 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:10.410950 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.410907 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:10.523550 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.523520 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd"] Jul 02 19:08:10.531296 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.531275 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd" Jul 02 19:08:10.542643 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.542625 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 02 19:08:10.543857 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.543812 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 02 19:08:10.543956 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.543914 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 02 19:08:10.544392 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.544365 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 02 19:08:10.544392 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.544389 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-p84kw\"" Jul 02 19:08:10.545941 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.545923 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd"] Jul 02 19:08:10.563302 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.563280 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:10.600389 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.600369 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd02a48a-bea1-4574-a1f7-3427ef175a1a-tmp\") pod \"kube-auth-proxy-7656b89f9d-2r6jd\" (UID: \"fd02a48a-bea1-4574-a1f7-3427ef175a1a\") " pod="openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd" Jul 02 19:08:10.600550 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.600405 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd02a48a-bea1-4574-a1f7-3427ef175a1a-tls-certs\") pod \"kube-auth-proxy-7656b89f9d-2r6jd\" (UID: \"fd02a48a-bea1-4574-a1f7-3427ef175a1a\") " pod="openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd" Jul 02 19:08:10.600550 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.600436 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlqmn\" (UniqueName: \"kubernetes.io/projected/fd02a48a-bea1-4574-a1f7-3427ef175a1a-kube-api-access-jlqmn\") pod \"kube-auth-proxy-7656b89f9d-2r6jd\" (UID: \"fd02a48a-bea1-4574-a1f7-3427ef175a1a\") " pod="openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd" Jul 02 19:08:10.659854 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.659820 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:10.700826 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.700774 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlqmn\" (UniqueName: \"kubernetes.io/projected/fd02a48a-bea1-4574-a1f7-3427ef175a1a-kube-api-access-jlqmn\") pod \"kube-auth-proxy-7656b89f9d-2r6jd\" (UID: \"fd02a48a-bea1-4574-a1f7-3427ef175a1a\") " pod="openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd" Jul 02 19:08:10.700920 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.700828 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd02a48a-bea1-4574-a1f7-3427ef175a1a-tmp\") pod \"kube-auth-proxy-7656b89f9d-2r6jd\" (UID: \"fd02a48a-bea1-4574-a1f7-3427ef175a1a\") " pod="openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd" Jul 02 19:08:10.700920 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.700885 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd02a48a-bea1-4574-a1f7-3427ef175a1a-tls-certs\") pod \"kube-auth-proxy-7656b89f9d-2r6jd\" (UID: \"fd02a48a-bea1-4574-a1f7-3427ef175a1a\") " pod="openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd" Jul 02 19:08:10.701024 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:08:10.700975 2565 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 02 19:08:10.701024 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:08:10.701024 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd02a48a-bea1-4574-a1f7-3427ef175a1a-tls-certs podName:fd02a48a-bea1-4574-a1f7-3427ef175a1a nodeName:}" failed. No retries permitted until 2026-07-02 19:08:11.20101005 +0000 UTC m=+320.997880219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fd02a48a-bea1-4574-a1f7-3427ef175a1a-tls-certs") pod "kube-auth-proxy-7656b89f9d-2r6jd" (UID: "fd02a48a-bea1-4574-a1f7-3427ef175a1a") : secret "kube-auth-proxy-tls" not found Jul 02 19:08:10.703065 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.703045 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd02a48a-bea1-4574-a1f7-3427ef175a1a-tmp\") pod \"kube-auth-proxy-7656b89f9d-2r6jd\" (UID: \"fd02a48a-bea1-4574-a1f7-3427ef175a1a\") " pod="openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd" Jul 02 19:08:10.716431 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.716402 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlqmn\" (UniqueName: \"kubernetes.io/projected/fd02a48a-bea1-4574-a1f7-3427ef175a1a-kube-api-access-jlqmn\") pod \"kube-auth-proxy-7656b89f9d-2r6jd\" (UID: \"fd02a48a-bea1-4574-a1f7-3427ef175a1a\") " pod="openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd" Jul 02 19:08:10.767391 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.767366 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:10.983122 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:10.983040 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:11.204176 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:11.204137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd02a48a-bea1-4574-a1f7-3427ef175a1a-tls-certs\") pod \"kube-auth-proxy-7656b89f9d-2r6jd\" (UID: \"fd02a48a-bea1-4574-a1f7-3427ef175a1a\") " pod="openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd" Jul 02 19:08:11.206771 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:11.206742 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd02a48a-bea1-4574-a1f7-3427ef175a1a-tls-certs\") pod \"kube-auth-proxy-7656b89f9d-2r6jd\" (UID: \"fd02a48a-bea1-4574-a1f7-3427ef175a1a\") " pod="openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd" Jul 02 19:08:11.276949 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:11.276919 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:11.401262 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:11.401232 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:11.440247 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:11.440223 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd" Jul 02 19:08:11.586976 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:11.586897 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd"] Jul 02 19:08:11.590283 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:08:11.590258 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd02a48a_bea1_4574_a1f7_3427ef175a1a.slice/crio-ec6151cc726f96f614c7f8f0fcca601062475e21443d0d646eac2ab23859dbe7 WatchSource:0}: Error finding container ec6151cc726f96f614c7f8f0fcca601062475e21443d0d646eac2ab23859dbe7: Status 404 returned error can't find the container with id ec6151cc726f96f614c7f8f0fcca601062475e21443d0d646eac2ab23859dbe7 Jul 02 19:08:12.075688 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:12.075654 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd" event={"ID":"fd02a48a-bea1-4574-a1f7-3427ef175a1a","Type":"ContainerStarted","Data":"ec6151cc726f96f614c7f8f0fcca601062475e21443d0d646eac2ab23859dbe7"} Jul 02 19:08:12.163003 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:12.162974 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:12.638810 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:12.638778 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:12.746245 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:12.746212 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:13.793582 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:13.793538 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:17.099025 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:17.098985 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd" event={"ID":"fd02a48a-bea1-4574-a1f7-3427ef175a1a","Type":"ContainerStarted","Data":"1f9a097ce24c0dabfe4aa14b56640d37e0b513c3b4d6504794ca39e7b289951a"} Jul 02 19:08:17.120672 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:17.120626 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7656b89f9d-2r6jd" podStartSLOduration=2.130722747 podStartE2EDuration="7.120611843s" totalImagesPullingTime="4.989889096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:17.120005753 +0000 UTC m=+326.916875946" watchObservedRunningTime="2026-07-02 19:08:17.120611843 +0000 UTC m=+326.917482034" Jul 02 19:08:18.533226 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.533195 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn"] Jul 02 19:08:18.536190 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.536173 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.546011 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.545994 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 02 19:08:18.546099 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.545995 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 02 19:08:18.547462 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.547449 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-s62hh\"" Jul 02 19:08:18.547769 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.547754 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 02 19:08:18.547806 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.547780 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 02 19:08:18.582786 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.582766 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn"] Jul 02 19:08:18.661464 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.661432 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvhc9\" (UniqueName: \"kubernetes.io/projected/6b4f8cf4-179c-43a6-9631-734bba0356cc-kube-api-access-tvhc9\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.661575 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.661470 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b4f8cf4-179c-43a6-9631-734bba0356cc-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.661575 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.661496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6b4f8cf4-179c-43a6-9631-734bba0356cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.661575 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.661537 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6b4f8cf4-179c-43a6-9631-734bba0356cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.661575 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.661555 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b4f8cf4-179c-43a6-9631-734bba0356cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.661705 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.661610 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b4f8cf4-179c-43a6-9631-734bba0356cc-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.661705 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.661632 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6b4f8cf4-179c-43a6-9631-734bba0356cc-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.764180 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.764153 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvhc9\" (UniqueName: \"kubernetes.io/projected/6b4f8cf4-179c-43a6-9631-734bba0356cc-kube-api-access-tvhc9\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.764304 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.764204 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b4f8cf4-179c-43a6-9631-734bba0356cc-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.764304 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.764266 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6b4f8cf4-179c-43a6-9631-734bba0356cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.764304 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.764300 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6b4f8cf4-179c-43a6-9631-734bba0356cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.764464 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.764329 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b4f8cf4-179c-43a6-9631-734bba0356cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.764464 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.764401 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b4f8cf4-179c-43a6-9631-734bba0356cc-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.764464 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.764450 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6b4f8cf4-179c-43a6-9631-734bba0356cc-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.765104 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.765080 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6b4f8cf4-179c-43a6-9631-734bba0356cc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.767102 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.767040 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6b4f8cf4-179c-43a6-9631-734bba0356cc-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.767102 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.767040 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b4f8cf4-179c-43a6-9631-734bba0356cc-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.767292 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.767112 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6b4f8cf4-179c-43a6-9631-734bba0356cc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.767402 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.767381 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b4f8cf4-179c-43a6-9631-734bba0356cc-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.772932 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.772913 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b4f8cf4-179c-43a6-9631-734bba0356cc-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.773011 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.772996 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvhc9\" (UniqueName: \"kubernetes.io/projected/6b4f8cf4-179c-43a6-9631-734bba0356cc-kube-api-access-tvhc9\") pod \"istiod-openshift-gateway-94bb8fbfd-zdhtn\" (UID: \"6b4f8cf4-179c-43a6-9631-734bba0356cc\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.844770 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.844717 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:18.992744 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:18.992722 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn"] Jul 02 19:08:18.994156 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:08:18.994130 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4f8cf4_179c_43a6_9631_734bba0356cc.slice/crio-c347a6ba433e3841d396d40234586b45b52492704c4d073b74444e70693653c1 WatchSource:0}: Error finding container c347a6ba433e3841d396d40234586b45b52492704c4d073b74444e70693653c1: Status 404 returned error can't find the container with id c347a6ba433e3841d396d40234586b45b52492704c4d073b74444e70693653c1 Jul 02 19:08:19.107556 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:19.107483 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" event={"ID":"6b4f8cf4-179c-43a6-9631-734bba0356cc","Type":"ContainerStarted","Data":"c347a6ba433e3841d396d40234586b45b52492704c4d073b74444e70693653c1"} Jul 02 19:08:20.231991 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.231952 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r"] Jul 02 19:08:20.236372 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.236357 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:20.240147 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.240117 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nwzwr\"" Jul 02 19:08:20.240285 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.240161 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 02 19:08:20.240285 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.240184 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 02 19:08:20.240285 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.240161 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 02 19:08:20.245246 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.245226 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r"] Jul 02 19:08:20.376910 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.376880 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/73a1c135-f1cc-4042-87ca-010ee4ab45b6-manager-config\") pod \"lws-controller-manager-6d578bc98c-8q47r\" (UID: \"73a1c135-f1cc-4042-87ca-010ee4ab45b6\") " pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:20.377063 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.376917 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/73a1c135-f1cc-4042-87ca-010ee4ab45b6-metrics-cert\") pod \"lws-controller-manager-6d578bc98c-8q47r\" (UID: \"73a1c135-f1cc-4042-87ca-010ee4ab45b6\") " pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:20.377063 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.376966 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73a1c135-f1cc-4042-87ca-010ee4ab45b6-cert\") pod \"lws-controller-manager-6d578bc98c-8q47r\" (UID: \"73a1c135-f1cc-4042-87ca-010ee4ab45b6\") " pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:20.377063 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.376985 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bl56\" (UniqueName: \"kubernetes.io/projected/73a1c135-f1cc-4042-87ca-010ee4ab45b6-kube-api-access-2bl56\") pod \"lws-controller-manager-6d578bc98c-8q47r\" (UID: \"73a1c135-f1cc-4042-87ca-010ee4ab45b6\") " pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:20.477504 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.477468 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73a1c135-f1cc-4042-87ca-010ee4ab45b6-cert\") pod \"lws-controller-manager-6d578bc98c-8q47r\" (UID: \"73a1c135-f1cc-4042-87ca-010ee4ab45b6\") " pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:20.477504 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.477506 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bl56\" (UniqueName: \"kubernetes.io/projected/73a1c135-f1cc-4042-87ca-010ee4ab45b6-kube-api-access-2bl56\") pod \"lws-controller-manager-6d578bc98c-8q47r\" (UID: \"73a1c135-f1cc-4042-87ca-010ee4ab45b6\") " pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:20.477744 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.477567 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/73a1c135-f1cc-4042-87ca-010ee4ab45b6-manager-config\") pod \"lws-controller-manager-6d578bc98c-8q47r\" (UID: \"73a1c135-f1cc-4042-87ca-010ee4ab45b6\") " pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:20.477744 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.477597 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/73a1c135-f1cc-4042-87ca-010ee4ab45b6-metrics-cert\") pod \"lws-controller-manager-6d578bc98c-8q47r\" (UID: \"73a1c135-f1cc-4042-87ca-010ee4ab45b6\") " pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:20.478322 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.478301 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/73a1c135-f1cc-4042-87ca-010ee4ab45b6-manager-config\") pod \"lws-controller-manager-6d578bc98c-8q47r\" (UID: \"73a1c135-f1cc-4042-87ca-010ee4ab45b6\") " pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:20.480062 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.480041 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/73a1c135-f1cc-4042-87ca-010ee4ab45b6-metrics-cert\") pod \"lws-controller-manager-6d578bc98c-8q47r\" (UID: \"73a1c135-f1cc-4042-87ca-010ee4ab45b6\") " pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:20.480163 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.480146 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73a1c135-f1cc-4042-87ca-010ee4ab45b6-cert\") pod \"lws-controller-manager-6d578bc98c-8q47r\" (UID: \"73a1c135-f1cc-4042-87ca-010ee4ab45b6\") " pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:20.513712 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.513693 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bl56\" (UniqueName: \"kubernetes.io/projected/73a1c135-f1cc-4042-87ca-010ee4ab45b6-kube-api-access-2bl56\") pod \"lws-controller-manager-6d578bc98c-8q47r\" (UID: \"73a1c135-f1cc-4042-87ca-010ee4ab45b6\") " pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:20.546006 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.545988 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:20.691519 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:20.691490 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r"] Jul 02 19:08:20.694469 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:08:20.694442 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a1c135_f1cc_4042_87ca_010ee4ab45b6.slice/crio-dc8135b3f1c5f29c758356d311ca532ab6a56d3a07e06fe7be6c95572c3b90e5 WatchSource:0}: Error finding container dc8135b3f1c5f29c758356d311ca532ab6a56d3a07e06fe7be6c95572c3b90e5: Status 404 returned error can't find the container with id dc8135b3f1c5f29c758356d311ca532ab6a56d3a07e06fe7be6c95572c3b90e5 Jul 02 19:08:21.117113 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:21.117074 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" event={"ID":"73a1c135-f1cc-4042-87ca-010ee4ab45b6","Type":"ContainerStarted","Data":"dc8135b3f1c5f29c758356d311ca532ab6a56d3a07e06fe7be6c95572c3b90e5"} Jul 02 19:08:21.396008 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:21.395924 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:21.508274 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:21.508235 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:21.642383 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:21.642350 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:22.034019 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:22.033987 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:22.673268 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:22.673215 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:22.780757 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:22.780723 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:22.965221 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:22.965132 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:23.160589 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:23.160554 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 19:08:23.160642 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:23.160623 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 02 19:08:23.856135 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:23.856103 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:23.977011 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:23.976978 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:24.133355 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:24.133260 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" event={"ID":"6b4f8cf4-179c-43a6-9631-734bba0356cc","Type":"ContainerStarted","Data":"a80d6b084fbbae19edafd175ac2978ada8f13408b3fd62ff2ec56d0b5c08975b"} Jul 02 19:08:24.133355 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:24.133350 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:24.135012 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:24.134963 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" event={"ID":"73a1c135-f1cc-4042-87ca-010ee4ab45b6","Type":"ContainerStarted","Data":"0b30b2347796ddc55846d494edec8a45237c7dd827afaeb8d8f56b9a90f01fa6"} Jul 02 19:08:24.135158 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:24.135046 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:24.135515 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:24.135487 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-zdhtn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 02 19:08:24.135611 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:24.135539 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" podUID="6b4f8cf4-179c-43a6-9631-734bba0356cc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:08:24.177128 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:24.177081 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" podStartSLOduration=1.7120782669999999 podStartE2EDuration="4.177066201s" totalImagesPullingTime="2.464987934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:24.175724107 +0000 UTC m=+333.972594298" watchObservedRunningTime="2026-07-02 19:08:24.177066201 +0000 UTC m=+333.973936392" Jul 02 19:08:24.177483 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:24.177463 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" podStartSLOduration=2.013311908 podStartE2EDuration="6.177456338s" totalImagesPullingTime="4.16414443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:08:24.159225862 +0000 UTC m=+333.956096054" watchObservedRunningTime="2026-07-02 19:08:24.177456338 +0000 UTC m=+333.974326551" Jul 02 19:08:24.776730 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:24.776697 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:24.920890 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:24.920837 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:25.033913 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:25.033825 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:25.139756 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:25.139730 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zdhtn" Jul 02 19:08:35.141449 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:35.141416 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6d578bc98c-8q47r" Jul 02 19:08:45.130635 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:45.130551 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:08:45.192766 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:08:45.192736 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:08.269883 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:08.269826 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:08.354733 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:08.354700 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:08.533617 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:08.533536 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:09.186409 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:09.186376 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:09.389517 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:09.389474 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:09.761527 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:09.761481 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:11.728109 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:11.728071 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:11.818288 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:11.818253 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:37.167000 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:37.166965 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:37.764229 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:37.764196 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:38.334153 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:38.334123 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:38.504628 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:38.504593 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:42.213070 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:42.213017 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:44.953385 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:44.953354 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:45.123871 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:45.123821 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:48.219824 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:48.219791 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:48.561888 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:48.561856 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:55.326474 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:55.326431 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:55.439895 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:55.439858 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:56.042864 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:56.042814 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:56.157740 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:56.157707 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:09:59.922691 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:09:59.922659 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:00.022208 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:00.022177 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:03.469807 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:03.469776 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:03.581226 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:03.581192 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:03.961007 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:03.960969 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:04.055856 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:04.055814 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:10.497790 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:10.497752 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:16.206189 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:16.206094 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:16.443800 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:16.443759 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:16.560855 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:16.560808 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:16.664122 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:16.664088 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:19.989699 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:19.989664 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:20.269888 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:20.269858 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:23.123063 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.123030 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8ffd9df7b-j28nh"] Jul 02 19:10:23.126159 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.126143 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.161647 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.161622 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8ffd9df7b-j28nh"] Jul 02 19:10:23.211016 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.210985 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:23.255128 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.255103 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-console-serving-cert\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.255235 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.255136 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpwcq\" (UniqueName: \"kubernetes.io/projected/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-kube-api-access-kpwcq\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.255235 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.255156 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-oauth-serving-cert\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.255235 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.255220 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-service-ca\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.255354 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.255250 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-console-config\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.255354 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.255299 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-trusted-ca-bundle\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.255354 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.255344 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-console-oauth-config\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.356006 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.355979 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-console-config\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.356115 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.356028 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-trusted-ca-bundle\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.356167 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.356146 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-console-oauth-config\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.356210 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.356197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-console-serving-cert\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.356254 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.356240 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpwcq\" (UniqueName: \"kubernetes.io/projected/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-kube-api-access-kpwcq\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.356290 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.356275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-oauth-serving-cert\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.356337 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.356323 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-service-ca\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.356810 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.356788 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-console-config\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.357102 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.356867 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-trusted-ca-bundle\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.357102 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.356939 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-service-ca\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.357223 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.357208 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-oauth-serving-cert\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.358711 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.358683 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-console-oauth-config\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.358939 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.358918 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-console-serving-cert\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.366366 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.366344 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpwcq\" (UniqueName: \"kubernetes.io/projected/65e458c3-9c44-4f8b-8dcb-911ffbcadf69-kube-api-access-kpwcq\") pod \"console-8ffd9df7b-j28nh\" (UID: \"65e458c3-9c44-4f8b-8dcb-911ffbcadf69\") " pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.395613 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.395560 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:23.434914 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.434888 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:23.589069 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.589040 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8ffd9df7b-j28nh"] Jul 02 19:10:23.591972 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:10:23.591951 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e458c3_9c44_4f8b_8dcb_911ffbcadf69.slice/crio-bd53be06c4abdb860252afece22a7edb02b8b33edfc5eff37696ba92db9059b7 WatchSource:0}: Error finding container bd53be06c4abdb860252afece22a7edb02b8b33edfc5eff37696ba92db9059b7: Status 404 returned error can't find the container with id bd53be06c4abdb860252afece22a7edb02b8b33edfc5eff37696ba92db9059b7 Jul 02 19:10:23.617061 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.617030 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8ffd9df7b-j28nh" event={"ID":"65e458c3-9c44-4f8b-8dcb-911ffbcadf69","Type":"ContainerStarted","Data":"bd53be06c4abdb860252afece22a7edb02b8b33edfc5eff37696ba92db9059b7"} Jul 02 19:10:23.787476 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.787439 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:23.896567 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:23.896541 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:24.622654 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:24.622608 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8ffd9df7b-j28nh" event={"ID":"65e458c3-9c44-4f8b-8dcb-911ffbcadf69","Type":"ContainerStarted","Data":"1690be8452f4b26fccdc4b50ab73cd15d2f9f9c4840d9210a2990863e944f5fa"} Jul 02 19:10:24.653678 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:24.653631 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8ffd9df7b-j28nh" podStartSLOduration=1.653616459 podStartE2EDuration="1.653616459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:10:24.652157572 +0000 UTC m=+454.449027788" watchObservedRunningTime="2026-07-02 19:10:24.653616459 +0000 UTC m=+454.450486706" Jul 02 19:10:27.242802 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:27.242754 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:27.394653 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:27.394613 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:33.436009 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:33.435969 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:33.436450 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:33.436046 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:33.440832 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:33.440810 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:33.662036 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:33.662010 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8ffd9df7b-j28nh" Jul 02 19:10:33.719222 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:33.719132 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cc866d7f9-kd9cx"] Jul 02 19:10:33.838399 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:33.838365 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:33.971060 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:33.970959 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:34.080458 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:34.080427 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:49.744436 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:49.744389 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:50.058163 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:50.058126 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:51.204087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:51.204057 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:55.723139 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:55.723106 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:55.820823 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:55.820794 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:55.902258 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:55.902225 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:56.593213 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:56.593173 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:56.693029 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:56.692998 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:58.067062 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:58.067026 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:10:58.740299 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:58.740246 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5cc866d7f9-kd9cx" podUID="6aad3414-2ec6-4b06-8a66-604aa95a55ca" containerName="console" containerID="cri-o://2e77d2f9fa076fac1478bbe0225be67b6dd41d2a37e81ff74b7d9885ea6654b2" gracePeriod=15 Jul 02 19:10:58.981174 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:58.981150 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:10:59.041690 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.041667 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-trusted-ca-bundle\") pod \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " Jul 02 19:10:59.041817 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.041701 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-service-ca\") pod \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " Jul 02 19:10:59.041817 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.041741 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-oauth-serving-cert\") pod \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " Jul 02 19:10:59.041817 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.041774 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-config\") pod \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " Jul 02 19:10:59.041817 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.041808 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-oauth-config\") pod \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " Jul 02 19:10:59.042024 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.041864 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-serving-cert\") pod \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " Jul 02 19:10:59.042024 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.041908 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc8n5\" (UniqueName: \"kubernetes.io/projected/6aad3414-2ec6-4b06-8a66-604aa95a55ca-kube-api-access-jc8n5\") pod \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\" (UID: \"6aad3414-2ec6-4b06-8a66-604aa95a55ca\") " Jul 02 19:10:59.042231 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.042086 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-service-ca" (OuterVolumeSpecName: "service-ca") pod "6aad3414-2ec6-4b06-8a66-604aa95a55ca" (UID: "6aad3414-2ec6-4b06-8a66-604aa95a55ca"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:10:59.042320 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.042210 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6aad3414-2ec6-4b06-8a66-604aa95a55ca" (UID: "6aad3414-2ec6-4b06-8a66-604aa95a55ca"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:10:59.042320 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.042274 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6aad3414-2ec6-4b06-8a66-604aa95a55ca" (UID: "6aad3414-2ec6-4b06-8a66-604aa95a55ca"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:10:59.042320 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.042284 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-config" (OuterVolumeSpecName: "console-config") pod "6aad3414-2ec6-4b06-8a66-604aa95a55ca" (UID: "6aad3414-2ec6-4b06-8a66-604aa95a55ca"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:10:59.044171 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.044143 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6aad3414-2ec6-4b06-8a66-604aa95a55ca" (UID: "6aad3414-2ec6-4b06-8a66-604aa95a55ca"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:10:59.044279 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.044193 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6aad3414-2ec6-4b06-8a66-604aa95a55ca" (UID: "6aad3414-2ec6-4b06-8a66-604aa95a55ca"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:10:59.044279 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.044251 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aad3414-2ec6-4b06-8a66-604aa95a55ca-kube-api-access-jc8n5" (OuterVolumeSpecName: "kube-api-access-jc8n5") pod "6aad3414-2ec6-4b06-8a66-604aa95a55ca" (UID: "6aad3414-2ec6-4b06-8a66-604aa95a55ca"). InnerVolumeSpecName "kube-api-access-jc8n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:10:59.142466 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.142440 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-trusted-ca-bundle\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:10:59.142466 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.142462 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:10:59.142466 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.142472 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-oauth-serving-cert\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:10:59.142822 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.142480 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-config\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:10:59.142822 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.142489 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-oauth-config\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:10:59.142822 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.142496 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6aad3414-2ec6-4b06-8a66-604aa95a55ca-console-serving-cert\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:10:59.142822 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.142504 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc8n5\" (UniqueName: \"kubernetes.io/projected/6aad3414-2ec6-4b06-8a66-604aa95a55ca-kube-api-access-jc8n5\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:10:59.768516 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.768477 2565 generic.go:356] "Generic (PLEG): container finished" podID="6aad3414-2ec6-4b06-8a66-604aa95a55ca" containerID="2e77d2f9fa076fac1478bbe0225be67b6dd41d2a37e81ff74b7d9885ea6654b2" exitCode=0 Jul 02 19:10:59.768516 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.768521 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cc866d7f9-kd9cx" Jul 02 19:10:59.768724 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.768558 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc866d7f9-kd9cx" event={"ID":"6aad3414-2ec6-4b06-8a66-604aa95a55ca","Type":"ContainerDied","Data":"2e77d2f9fa076fac1478bbe0225be67b6dd41d2a37e81ff74b7d9885ea6654b2"} Jul 02 19:10:59.768724 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.768602 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cc866d7f9-kd9cx" event={"ID":"6aad3414-2ec6-4b06-8a66-604aa95a55ca","Type":"ContainerDied","Data":"9c5de005411648c8ff0279cc0744333d0b27b5475eee1fcb7879e45a274edcad"} Jul 02 19:10:59.768724 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.768622 2565 scope.go:117] "RemoveContainer" containerID="2e77d2f9fa076fac1478bbe0225be67b6dd41d2a37e81ff74b7d9885ea6654b2" Jul 02 19:10:59.777346 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.777331 2565 scope.go:117] "RemoveContainer" containerID="2e77d2f9fa076fac1478bbe0225be67b6dd41d2a37e81ff74b7d9885ea6654b2" Jul 02 19:10:59.777586 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:10:59.777565 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e77d2f9fa076fac1478bbe0225be67b6dd41d2a37e81ff74b7d9885ea6654b2\": container with ID starting with 2e77d2f9fa076fac1478bbe0225be67b6dd41d2a37e81ff74b7d9885ea6654b2 not found: ID does not exist" containerID="2e77d2f9fa076fac1478bbe0225be67b6dd41d2a37e81ff74b7d9885ea6654b2" Jul 02 19:10:59.777649 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.777597 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e77d2f9fa076fac1478bbe0225be67b6dd41d2a37e81ff74b7d9885ea6654b2"} err="failed to get container status \"2e77d2f9fa076fac1478bbe0225be67b6dd41d2a37e81ff74b7d9885ea6654b2\": rpc error: code = NotFound desc = could not find container \"2e77d2f9fa076fac1478bbe0225be67b6dd41d2a37e81ff74b7d9885ea6654b2\": container with ID starting with 2e77d2f9fa076fac1478bbe0225be67b6dd41d2a37e81ff74b7d9885ea6654b2 not found: ID does not exist" Jul 02 19:10:59.789157 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.789135 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cc866d7f9-kd9cx"] Jul 02 19:10:59.795074 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:10:59.795056 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cc866d7f9-kd9cx"] Jul 02 19:11:00.763005 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:11:00.762970 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aad3414-2ec6-4b06-8a66-604aa95a55ca" path="/var/lib/kubelet/pods/6aad3414-2ec6-4b06-8a66-604aa95a55ca/volumes" Jul 02 19:11:12.321450 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:11:12.321418 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:25.109227 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:11:25.109191 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:25.212830 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:11:25.212799 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:27.058143 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:11:27.058108 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:30.744780 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:11:30.744746 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:31.661814 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:11:31.661767 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:31.763334 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:11:31.763304 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:31.976085 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:11:31.976002 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:34.559717 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:11:34.559675 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:11:34.670022 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:11:34.669990 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:07.160271 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:07.160236 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:07.633178 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:07.633148 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:08.124047 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:08.124011 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:08.211562 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:08.211535 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:19.213057 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:19.213021 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:20.752966 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:20.752928 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:26.814864 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:26.814816 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:34.705541 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:34.705510 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:47.838548 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:47.838514 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:51.133358 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.133257 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:12:51.137439 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.135347 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:12:51.305748 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.305723 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv"] Jul 02 19:12:51.306058 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.306045 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aad3414-2ec6-4b06-8a66-604aa95a55ca" containerName="console" Jul 02 19:12:51.306105 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.306061 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aad3414-2ec6-4b06-8a66-604aa95a55ca" containerName="console" Jul 02 19:12:51.306142 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.306106 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aad3414-2ec6-4b06-8a66-604aa95a55ca" containerName="console" Jul 02 19:12:51.308756 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.308739 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.314572 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.314550 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 02 19:12:51.314701 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.314577 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 02 19:12:51.314701 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.314601 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 02 19:12:51.314701 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.314609 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mwzqz\"" Jul 02 19:12:51.325420 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.325400 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv"] Jul 02 19:12:51.366098 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.366070 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.366197 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.366106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z4q5\" (UniqueName: \"kubernetes.io/projected/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-kube-api-access-4z4q5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.366197 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.366133 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.366197 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.366151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.366297 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.366227 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.366297 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.366259 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.373980 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.373947 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:12:51.466790 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.466691 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.466790 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.466739 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.466790 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.466788 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.467078 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.466815 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.467078 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.466910 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.467078 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.466952 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z4q5\" (UniqueName: \"kubernetes.io/projected/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-kube-api-access-4z4q5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.467289 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.467083 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.467463 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.467426 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.467554 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.467532 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.469827 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.469805 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.469946 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.469897 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.474741 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.474715 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z4q5\" (UniqueName: \"kubernetes.io/projected/90442dac-8e2f-4d15-a56c-2d0c6ef52cb8-kube-api-access-4z4q5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv\" (UID: \"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.619008 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.618968 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:12:51.750142 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:51.750112 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv"] Jul 02 19:12:51.752238 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:12:51.752215 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90442dac_8e2f_4d15_a56c_2d0c6ef52cb8.slice/crio-14559be496015eaa11b6dd38c9e96b54ed17c952ddf1f30f27227d41198065f1 WatchSource:0}: Error finding container 14559be496015eaa11b6dd38c9e96b54ed17c952ddf1f30f27227d41198065f1: Status 404 returned error can't find the container with id 14559be496015eaa11b6dd38c9e96b54ed17c952ddf1f30f27227d41198065f1 Jul 02 19:12:52.224357 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:52.224218 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" event={"ID":"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8","Type":"ContainerStarted","Data":"14559be496015eaa11b6dd38c9e96b54ed17c952ddf1f30f27227d41198065f1"} Jul 02 19:12:53.230631 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:53.230538 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" event={"ID":"90442dac-8e2f-4d15-a56c-2d0c6ef52cb8","Type":"ContainerStarted","Data":"3aa741c8265a6bb82ba6b0cfa180478bb4d384b770d508f672613d639c4a700c"} Jul 02 19:12:53.253915 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:12:53.253866 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" podStartSLOduration=1.205522757 podStartE2EDuration="2.253835425s" totalImagesPullingTime="1.048312668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:12:53.252335162 +0000 UTC m=+603.049205353" watchObservedRunningTime="2026-07-02 19:12:53.253835425 +0000 UTC m=+603.050705616" Jul 02 19:13:01.620100 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:13:01.620058 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:13:01.620572 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:13:01.620127 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:13:01.632426 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:13:01.632400 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:13:02.276927 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:13:02.276895 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv" Jul 02 19:13:02.519550 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:13:02.519504 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:11.906501 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:13:11.906409 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:18.642757 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:13:18.642725 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:13:52.471978 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:13:52.471940 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:14:35.624007 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:14:35.623970 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:14:35.722523 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:14:35.722487 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:14:36.910734 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:14:36.910701 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:14:37.050029 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:14:37.049993 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:40.724704 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:15:40.724664 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:48.116364 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:15:48.116328 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:15:50.087742 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:15:50.087707 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:23.807716 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:16:23.807684 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:23.906822 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:16:23.906790 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:24.030066 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:16:24.030032 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:29.835431 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:16:29.835401 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:29.945296 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:16:29.945266 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:30.051082 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:16:30.051048 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:30.194902 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:16:30.194803 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:35.576307 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:16:35.576276 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:35.666954 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:16:35.666920 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:36.263053 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:16:36.263021 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:16:36.359469 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:16:36.359437 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:17:24.992860 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:17:24.992806 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:17:51.162427 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:17:51.162315 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:17:51.168398 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:17:51.165101 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:20:24.780630 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:20:24.780596 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:20:24.841469 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:20:24.841438 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:22:17.065327 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:22:17.065291 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:22:51.187333 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:22:51.187223 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:22:51.190635 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:22:51.188957 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:25:28.512238 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:25:28.512203 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:09.831699 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:09.831603 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:09.920580 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:09.920550 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:17.301672 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:17.301636 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:25.573990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:25.573953 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:30.709437 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:30.709394 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:33.629041 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:33.629005 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:33.719337 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:33.719299 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:34.082800 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:34.082762 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:39.692894 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:39.692855 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:39.794632 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:39.794603 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:39.909865 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:39.909792 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:40.014241 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:40.014212 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:41.120464 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.120433 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:41.198319 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.198266 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z"] Jul 02 19:27:41.201578 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.201554 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" Jul 02 19:27:41.204730 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.204708 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-b0830766-serving-cert\"" Jul 02 19:27:41.204878 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.204783 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxm9f\"" Jul 02 19:27:41.211985 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.211961 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z"] Jul 02 19:27:41.229720 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.229688 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:41.311880 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.311818 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90641083-14c4-4695-b5c1-34a3341ed6c1-maas-api-tls\") pod \"maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z\" (UID: \"90641083-14c4-4695-b5c1-34a3341ed6c1\") " pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" Jul 02 19:27:41.312037 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.312007 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90641083-14c4-4695-b5c1-34a3341ed6c1-openshift-service-ca\") pod \"maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z\" (UID: \"90641083-14c4-4695-b5c1-34a3341ed6c1\") " pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" Jul 02 19:27:41.312107 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.312049 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqnnt\" (UniqueName: \"kubernetes.io/projected/90641083-14c4-4695-b5c1-34a3341ed6c1-kube-api-access-pqnnt\") pod \"maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z\" (UID: \"90641083-14c4-4695-b5c1-34a3341ed6c1\") " pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" Jul 02 19:27:41.322489 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.322465 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:41.412647 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.412581 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90641083-14c4-4695-b5c1-34a3341ed6c1-maas-api-tls\") pod \"maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z\" (UID: \"90641083-14c4-4695-b5c1-34a3341ed6c1\") " pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" Jul 02 19:27:41.412647 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.412632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90641083-14c4-4695-b5c1-34a3341ed6c1-openshift-service-ca\") pod \"maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z\" (UID: \"90641083-14c4-4695-b5c1-34a3341ed6c1\") " pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" Jul 02 19:27:41.412868 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.412658 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqnnt\" (UniqueName: \"kubernetes.io/projected/90641083-14c4-4695-b5c1-34a3341ed6c1-kube-api-access-pqnnt\") pod \"maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z\" (UID: \"90641083-14c4-4695-b5c1-34a3341ed6c1\") " pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" Jul 02 19:27:41.412868 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:27:41.412748 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-b0830766-serving-cert: secret "maas-api-e2e-ait-b0830766-serving-cert" not found Jul 02 19:27:41.412868 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:27:41.412864 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90641083-14c4-4695-b5c1-34a3341ed6c1-maas-api-tls podName:90641083-14c4-4695-b5c1-34a3341ed6c1 nodeName:}" failed. No retries permitted until 2026-07-02 19:27:41.912815865 +0000 UTC m=+1491.709686051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/90641083-14c4-4695-b5c1-34a3341ed6c1-maas-api-tls") pod "maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" (UID: "90641083-14c4-4695-b5c1-34a3341ed6c1") : secret "maas-api-e2e-ait-b0830766-serving-cert" not found Jul 02 19:27:41.413291 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.413273 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90641083-14c4-4695-b5c1-34a3341ed6c1-openshift-service-ca\") pod \"maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z\" (UID: \"90641083-14c4-4695-b5c1-34a3341ed6c1\") " pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" Jul 02 19:27:41.423211 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.423193 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqnnt\" (UniqueName: \"kubernetes.io/projected/90641083-14c4-4695-b5c1-34a3341ed6c1-kube-api-access-pqnnt\") pod \"maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z\" (UID: \"90641083-14c4-4695-b5c1-34a3341ed6c1\") " pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" Jul 02 19:27:41.524218 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.524193 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:41.788780 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.788747 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:41.892002 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.891970 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:41.917277 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.917248 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90641083-14c4-4695-b5c1-34a3341ed6c1-maas-api-tls\") pod \"maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z\" (UID: \"90641083-14c4-4695-b5c1-34a3341ed6c1\") " pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" Jul 02 19:27:41.919992 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:41.919968 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90641083-14c4-4695-b5c1-34a3341ed6c1-maas-api-tls\") pod \"maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z\" (UID: \"90641083-14c4-4695-b5c1-34a3341ed6c1\") " pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" Jul 02 19:27:42.113588 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:42.113496 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" Jul 02 19:27:42.242442 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:42.242420 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z"] Jul 02 19:27:42.244760 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:27:42.244731 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90641083_14c4_4695_b5c1_34a3341ed6c1.slice/crio-7b87ebc8b74dcf3d995e2609b862c93ea60c6dfc7f663ef87985a50cd91687d3 WatchSource:0}: Error finding container 7b87ebc8b74dcf3d995e2609b862c93ea60c6dfc7f663ef87985a50cd91687d3: Status 404 returned error can't find the container with id 7b87ebc8b74dcf3d995e2609b862c93ea60c6dfc7f663ef87985a50cd91687d3 Jul 02 19:27:42.246188 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:42.246167 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:27:42.925606 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:42.925573 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" event={"ID":"90641083-14c4-4695-b5c1-34a3341ed6c1","Type":"ContainerStarted","Data":"7b87ebc8b74dcf3d995e2609b862c93ea60c6dfc7f663ef87985a50cd91687d3"} Jul 02 19:27:43.693153 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:43.693119 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:44.936602 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:44.936517 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" event={"ID":"90641083-14c4-4695-b5c1-34a3341ed6c1","Type":"ContainerStarted","Data":"46e5bbbcccd97188ba243037515e079d5cc009b43c4bfee069ea364694b6d6e4"} Jul 02 19:27:44.936986 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:44.936657 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" Jul 02 19:27:44.953989 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:44.953939 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" podStartSLOduration=1.5244817510000002 podStartE2EDuration="3.953927514s" totalImagesPullingTime="2.429445763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:27:44.952941092 +0000 UTC m=+1494.749811309" watchObservedRunningTime="2026-07-02 19:27:44.953927514 +0000 UTC m=+1494.750797704" Jul 02 19:27:46.381690 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:46.381656 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:46.522147 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:46.522115 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:46.627328 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:46.627288 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:46.709773 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:46.709695 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z"] Jul 02 19:27:46.741043 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:46.741019 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:46.944560 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:46.944514 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" podUID="90641083-14c4-4695-b5c1-34a3341ed6c1" containerName="maas-api" containerID="cri-o://46e5bbbcccd97188ba243037515e079d5cc009b43c4bfee069ea364694b6d6e4" gracePeriod=30 Jul 02 19:27:47.184141 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.184110 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" Jul 02 19:27:47.267237 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.267208 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90641083-14c4-4695-b5c1-34a3341ed6c1-maas-api-tls\") pod \"90641083-14c4-4695-b5c1-34a3341ed6c1\" (UID: \"90641083-14c4-4695-b5c1-34a3341ed6c1\") " Jul 02 19:27:47.267393 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.267258 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqnnt\" (UniqueName: \"kubernetes.io/projected/90641083-14c4-4695-b5c1-34a3341ed6c1-kube-api-access-pqnnt\") pod \"90641083-14c4-4695-b5c1-34a3341ed6c1\" (UID: \"90641083-14c4-4695-b5c1-34a3341ed6c1\") " Jul 02 19:27:47.267393 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.267303 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90641083-14c4-4695-b5c1-34a3341ed6c1-openshift-service-ca\") pod \"90641083-14c4-4695-b5c1-34a3341ed6c1\" (UID: \"90641083-14c4-4695-b5c1-34a3341ed6c1\") " Jul 02 19:27:47.267699 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.267675 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90641083-14c4-4695-b5c1-34a3341ed6c1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "90641083-14c4-4695-b5c1-34a3341ed6c1" (UID: "90641083-14c4-4695-b5c1-34a3341ed6c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:27:47.269560 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.269538 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90641083-14c4-4695-b5c1-34a3341ed6c1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "90641083-14c4-4695-b5c1-34a3341ed6c1" (UID: "90641083-14c4-4695-b5c1-34a3341ed6c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:27:47.269641 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.269592 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90641083-14c4-4695-b5c1-34a3341ed6c1-kube-api-access-pqnnt" (OuterVolumeSpecName: "kube-api-access-pqnnt") pod "90641083-14c4-4695-b5c1-34a3341ed6c1" (UID: "90641083-14c4-4695-b5c1-34a3341ed6c1"). InnerVolumeSpecName "kube-api-access-pqnnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:27:47.368704 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.368642 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90641083-14c4-4695-b5c1-34a3341ed6c1-openshift-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:27:47.368704 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.368668 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90641083-14c4-4695-b5c1-34a3341ed6c1-maas-api-tls\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:27:47.368704 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.368677 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqnnt\" (UniqueName: \"kubernetes.io/projected/90641083-14c4-4695-b5c1-34a3341ed6c1-kube-api-access-pqnnt\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:27:47.950104 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.950067 2565 generic.go:356] "Generic (PLEG): container finished" podID="90641083-14c4-4695-b5c1-34a3341ed6c1" containerID="46e5bbbcccd97188ba243037515e079d5cc009b43c4bfee069ea364694b6d6e4" exitCode=0 Jul 02 19:27:47.950472 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.950117 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" Jul 02 19:27:47.950472 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.950121 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" event={"ID":"90641083-14c4-4695-b5c1-34a3341ed6c1","Type":"ContainerDied","Data":"46e5bbbcccd97188ba243037515e079d5cc009b43c4bfee069ea364694b6d6e4"} Jul 02 19:27:47.950472 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.950165 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z" event={"ID":"90641083-14c4-4695-b5c1-34a3341ed6c1","Type":"ContainerDied","Data":"7b87ebc8b74dcf3d995e2609b862c93ea60c6dfc7f663ef87985a50cd91687d3"} Jul 02 19:27:47.950472 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.950184 2565 scope.go:117] "RemoveContainer" containerID="46e5bbbcccd97188ba243037515e079d5cc009b43c4bfee069ea364694b6d6e4" Jul 02 19:27:47.959421 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.959403 2565 scope.go:117] "RemoveContainer" containerID="46e5bbbcccd97188ba243037515e079d5cc009b43c4bfee069ea364694b6d6e4" Jul 02 19:27:47.959654 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:27:47.959634 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e5bbbcccd97188ba243037515e079d5cc009b43c4bfee069ea364694b6d6e4\": container with ID starting with 46e5bbbcccd97188ba243037515e079d5cc009b43c4bfee069ea364694b6d6e4 not found: ID does not exist" containerID="46e5bbbcccd97188ba243037515e079d5cc009b43c4bfee069ea364694b6d6e4" Jul 02 19:27:47.959696 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.959669 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e5bbbcccd97188ba243037515e079d5cc009b43c4bfee069ea364694b6d6e4"} err="failed to get container status \"46e5bbbcccd97188ba243037515e079d5cc009b43c4bfee069ea364694b6d6e4\": rpc error: code = NotFound desc = could not find container \"46e5bbbcccd97188ba243037515e079d5cc009b43c4bfee069ea364694b6d6e4\": container with ID starting with 46e5bbbcccd97188ba243037515e079d5cc009b43c4bfee069ea364694b6d6e4 not found: ID does not exist" Jul 02 19:27:47.988753 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:47.988732 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z"] Jul 02 19:27:48.005123 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:48.005099 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-b0830766-7c9cff9fb7-n2r7z"] Jul 02 19:27:48.342541 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:48.342512 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:48.762775 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:48.762744 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90641083-14c4-4695-b5c1-34a3341ed6c1" path="/var/lib/kubelet/pods/90641083-14c4-4695-b5c1-34a3341ed6c1/volumes" Jul 02 19:27:51.216057 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:51.215899 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:27:51.232306 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:51.216905 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:27:52.278332 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:52.278298 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:52.378168 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:52.378138 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:52.493082 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:52.493051 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:52.586598 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:52.586470 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:53.683005 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:53.682970 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:53.741926 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:53.741897 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5"] Jul 02 19:27:53.742259 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:53.742242 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90641083-14c4-4695-b5c1-34a3341ed6c1" containerName="maas-api" Jul 02 19:27:53.742400 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:53.742265 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="90641083-14c4-4695-b5c1-34a3341ed6c1" containerName="maas-api" Jul 02 19:27:53.742400 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:53.742344 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="90641083-14c4-4695-b5c1-34a3341ed6c1" containerName="maas-api" Jul 02 19:27:53.745355 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:53.745336 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" Jul 02 19:27:53.747826 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:53.747806 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxm9f\"" Jul 02 19:27:53.747968 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:53.747805 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-74a34470-serving-cert\"" Jul 02 19:27:53.756898 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:53.756875 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5"] Jul 02 19:27:53.771478 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:53.771450 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:53.840311 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:53.840284 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:53.925221 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:53.925192 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc48f97a-eff6-4125-b282-9adc30627072-openshift-service-ca\") pod \"maas-api-e2e-ait-74a34470-c6d766b65-qvxd5\" (UID: \"dc48f97a-eff6-4125-b282-9adc30627072\") " pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" Jul 02 19:27:53.925381 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:53.925299 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj7qs\" (UniqueName: \"kubernetes.io/projected/dc48f97a-eff6-4125-b282-9adc30627072-kube-api-access-jj7qs\") pod \"maas-api-e2e-ait-74a34470-c6d766b65-qvxd5\" (UID: \"dc48f97a-eff6-4125-b282-9adc30627072\") " pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" Jul 02 19:27:53.925381 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:53.925353 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc48f97a-eff6-4125-b282-9adc30627072-maas-api-tls\") pod \"maas-api-e2e-ait-74a34470-c6d766b65-qvxd5\" (UID: \"dc48f97a-eff6-4125-b282-9adc30627072\") " pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" Jul 02 19:27:54.026433 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:54.026407 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj7qs\" (UniqueName: \"kubernetes.io/projected/dc48f97a-eff6-4125-b282-9adc30627072-kube-api-access-jj7qs\") pod \"maas-api-e2e-ait-74a34470-c6d766b65-qvxd5\" (UID: \"dc48f97a-eff6-4125-b282-9adc30627072\") " pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" Jul 02 19:27:54.026578 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:54.026464 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc48f97a-eff6-4125-b282-9adc30627072-maas-api-tls\") pod \"maas-api-e2e-ait-74a34470-c6d766b65-qvxd5\" (UID: \"dc48f97a-eff6-4125-b282-9adc30627072\") " pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" Jul 02 19:27:54.026578 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:54.026491 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc48f97a-eff6-4125-b282-9adc30627072-openshift-service-ca\") pod \"maas-api-e2e-ait-74a34470-c6d766b65-qvxd5\" (UID: \"dc48f97a-eff6-4125-b282-9adc30627072\") " pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" Jul 02 19:27:54.027155 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:54.027134 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc48f97a-eff6-4125-b282-9adc30627072-openshift-service-ca\") pod \"maas-api-e2e-ait-74a34470-c6d766b65-qvxd5\" (UID: \"dc48f97a-eff6-4125-b282-9adc30627072\") " pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" Jul 02 19:27:54.029110 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:54.029092 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc48f97a-eff6-4125-b282-9adc30627072-maas-api-tls\") pod \"maas-api-e2e-ait-74a34470-c6d766b65-qvxd5\" (UID: \"dc48f97a-eff6-4125-b282-9adc30627072\") " pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" Jul 02 19:27:54.040180 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:54.040156 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj7qs\" (UniqueName: \"kubernetes.io/projected/dc48f97a-eff6-4125-b282-9adc30627072-kube-api-access-jj7qs\") pod \"maas-api-e2e-ait-74a34470-c6d766b65-qvxd5\" (UID: \"dc48f97a-eff6-4125-b282-9adc30627072\") " pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" Jul 02 19:27:54.041567 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:54.041544 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:54.056814 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:54.056789 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" Jul 02 19:27:54.226400 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:54.226366 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5"] Jul 02 19:27:54.235629 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:27:54.235594 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc48f97a_eff6_4125_b282_9adc30627072.slice/crio-bc8ea30b4379ea371124146d8d7b8cb112b21897ac66a4ee2eee4c14184705fe WatchSource:0}: Error finding container bc8ea30b4379ea371124146d8d7b8cb112b21897ac66a4ee2eee4c14184705fe: Status 404 returned error can't find the container with id bc8ea30b4379ea371124146d8d7b8cb112b21897ac66a4ee2eee4c14184705fe Jul 02 19:27:54.492069 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:54.491968 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:54.592275 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:54.592224 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:54.983988 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:54.983953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" event={"ID":"dc48f97a-eff6-4125-b282-9adc30627072","Type":"ContainerStarted","Data":"bd4d63e473b726cac0d891b6d757db0e52a3f1077beefee03c1242a598f2f3a1"} Jul 02 19:27:54.984381 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:54.984003 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" event={"ID":"dc48f97a-eff6-4125-b282-9adc30627072","Type":"ContainerStarted","Data":"bc8ea30b4379ea371124146d8d7b8cb112b21897ac66a4ee2eee4c14184705fe"} Jul 02 19:27:54.984381 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:54.984042 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" Jul 02 19:27:55.001762 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:55.001714 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" podStartSLOduration=1.675709468 podStartE2EDuration="2.001698765s" totalImagesPullingTime="325.989297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:27:55.000101714 +0000 UTC m=+1504.796971951" watchObservedRunningTime="2026-07-02 19:27:55.001698765 +0000 UTC m=+1504.798568956" Jul 02 19:27:55.767188 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:55.767153 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:58.848268 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:58.848234 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:59.159222 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:59.159147 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5"] Jul 02 19:27:59.159389 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:59.159362 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" podUID="dc48f97a-eff6-4125-b282-9adc30627072" containerName="maas-api" containerID="cri-o://bd4d63e473b726cac0d891b6d757db0e52a3f1077beefee03c1242a598f2f3a1" gracePeriod=30 Jul 02 19:27:59.165570 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:59.165551 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" Jul 02 19:27:59.187276 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:59.187256 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:27:59.400222 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:59.400201 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" Jul 02 19:27:59.472039 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:59.471957 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc48f97a-eff6-4125-b282-9adc30627072-maas-api-tls\") pod \"dc48f97a-eff6-4125-b282-9adc30627072\" (UID: \"dc48f97a-eff6-4125-b282-9adc30627072\") " Jul 02 19:27:59.472039 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:59.471988 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj7qs\" (UniqueName: \"kubernetes.io/projected/dc48f97a-eff6-4125-b282-9adc30627072-kube-api-access-jj7qs\") pod \"dc48f97a-eff6-4125-b282-9adc30627072\" (UID: \"dc48f97a-eff6-4125-b282-9adc30627072\") " Jul 02 19:27:59.472039 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:59.472027 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc48f97a-eff6-4125-b282-9adc30627072-openshift-service-ca\") pod \"dc48f97a-eff6-4125-b282-9adc30627072\" (UID: \"dc48f97a-eff6-4125-b282-9adc30627072\") " Jul 02 19:27:59.472409 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:59.472385 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc48f97a-eff6-4125-b282-9adc30627072-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc48f97a-eff6-4125-b282-9adc30627072" (UID: "dc48f97a-eff6-4125-b282-9adc30627072"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:27:59.474204 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:59.474182 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc48f97a-eff6-4125-b282-9adc30627072-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc48f97a-eff6-4125-b282-9adc30627072" (UID: "dc48f97a-eff6-4125-b282-9adc30627072"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:27:59.474293 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:59.474223 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc48f97a-eff6-4125-b282-9adc30627072-kube-api-access-jj7qs" (OuterVolumeSpecName: "kube-api-access-jj7qs") pod "dc48f97a-eff6-4125-b282-9adc30627072" (UID: "dc48f97a-eff6-4125-b282-9adc30627072"). InnerVolumeSpecName "kube-api-access-jj7qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:27:59.573391 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:59.573360 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc48f97a-eff6-4125-b282-9adc30627072-maas-api-tls\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:27:59.573391 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:59.573386 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj7qs\" (UniqueName: \"kubernetes.io/projected/dc48f97a-eff6-4125-b282-9adc30627072-kube-api-access-jj7qs\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:27:59.573391 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:27:59.573395 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc48f97a-eff6-4125-b282-9adc30627072-openshift-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:28:00.007247 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:00.007210 2565 generic.go:356] "Generic (PLEG): container finished" podID="dc48f97a-eff6-4125-b282-9adc30627072" containerID="bd4d63e473b726cac0d891b6d757db0e52a3f1077beefee03c1242a598f2f3a1" exitCode=0 Jul 02 19:28:00.007704 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:00.007256 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" Jul 02 19:28:00.007704 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:00.007262 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" event={"ID":"dc48f97a-eff6-4125-b282-9adc30627072","Type":"ContainerDied","Data":"bd4d63e473b726cac0d891b6d757db0e52a3f1077beefee03c1242a598f2f3a1"} Jul 02 19:28:00.007704 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:00.007316 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5" event={"ID":"dc48f97a-eff6-4125-b282-9adc30627072","Type":"ContainerDied","Data":"bc8ea30b4379ea371124146d8d7b8cb112b21897ac66a4ee2eee4c14184705fe"} Jul 02 19:28:00.007704 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:00.007348 2565 scope.go:117] "RemoveContainer" containerID="bd4d63e473b726cac0d891b6d757db0e52a3f1077beefee03c1242a598f2f3a1" Jul 02 19:28:00.015826 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:00.015807 2565 scope.go:117] "RemoveContainer" containerID="bd4d63e473b726cac0d891b6d757db0e52a3f1077beefee03c1242a598f2f3a1" Jul 02 19:28:00.016101 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:28:00.016082 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4d63e473b726cac0d891b6d757db0e52a3f1077beefee03c1242a598f2f3a1\": container with ID starting with bd4d63e473b726cac0d891b6d757db0e52a3f1077beefee03c1242a598f2f3a1 not found: ID does not exist" containerID="bd4d63e473b726cac0d891b6d757db0e52a3f1077beefee03c1242a598f2f3a1" Jul 02 19:28:00.016155 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:00.016115 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4d63e473b726cac0d891b6d757db0e52a3f1077beefee03c1242a598f2f3a1"} err="failed to get container status \"bd4d63e473b726cac0d891b6d757db0e52a3f1077beefee03c1242a598f2f3a1\": rpc error: code = NotFound desc = could not find container \"bd4d63e473b726cac0d891b6d757db0e52a3f1077beefee03c1242a598f2f3a1\": container with ID starting with bd4d63e473b726cac0d891b6d757db0e52a3f1077beefee03c1242a598f2f3a1 not found: ID does not exist" Jul 02 19:28:00.027304 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:00.027282 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5"] Jul 02 19:28:00.030920 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:00.030899 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-74a34470-c6d766b65-qvxd5"] Jul 02 19:28:00.763074 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:00.763041 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc48f97a-eff6-4125-b282-9adc30627072" path="/var/lib/kubelet/pods/dc48f97a-eff6-4125-b282-9adc30627072/volumes" Jul 02 19:28:05.279121 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:05.279090 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:05.386945 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:05.386914 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:06.929536 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:06.929508 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:11.350058 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:11.349986 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:11.427391 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:11.427333 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:11.574217 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:11.574182 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:11.645652 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:11.645578 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:12.781658 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.781623 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:12.847523 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.847494 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556"] Jul 02 19:28:12.847873 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.847861 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc48f97a-eff6-4125-b282-9adc30627072" containerName="maas-api" Jul 02 19:28:12.847933 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.847876 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc48f97a-eff6-4125-b282-9adc30627072" containerName="maas-api" Jul 02 19:28:12.847980 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.847949 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc48f97a-eff6-4125-b282-9adc30627072" containerName="maas-api" Jul 02 19:28:12.852267 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.852247 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" Jul 02 19:28:12.854613 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.854587 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-079596c4-serving-cert\"" Jul 02 19:28:12.854730 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.854707 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxm9f\"" Jul 02 19:28:12.860404 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.860385 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556"] Jul 02 19:28:12.863546 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.863519 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:12.886523 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.886495 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c4fe6-5eec-4971-812e-59fac0d832b1-openshift-service-ca\") pod \"maas-api-e2e-derive-079596c4-65cc46fbd9-z7556\" (UID: \"195c4fe6-5eec-4971-812e-59fac0d832b1\") " pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" Jul 02 19:28:12.886654 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.886530 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195c4fe6-5eec-4971-812e-59fac0d832b1-maas-api-tls\") pod \"maas-api-e2e-derive-079596c4-65cc46fbd9-z7556\" (UID: \"195c4fe6-5eec-4971-812e-59fac0d832b1\") " pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" Jul 02 19:28:12.886654 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.886624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wv7r\" (UniqueName: \"kubernetes.io/projected/195c4fe6-5eec-4971-812e-59fac0d832b1-kube-api-access-4wv7r\") pod \"maas-api-e2e-derive-079596c4-65cc46fbd9-z7556\" (UID: \"195c4fe6-5eec-4971-812e-59fac0d832b1\") " pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" Jul 02 19:28:12.941828 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.941799 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:12.987932 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.987904 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c4fe6-5eec-4971-812e-59fac0d832b1-openshift-service-ca\") pod \"maas-api-e2e-derive-079596c4-65cc46fbd9-z7556\" (UID: \"195c4fe6-5eec-4971-812e-59fac0d832b1\") " pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" Jul 02 19:28:12.988092 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.987938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195c4fe6-5eec-4971-812e-59fac0d832b1-maas-api-tls\") pod \"maas-api-e2e-derive-079596c4-65cc46fbd9-z7556\" (UID: \"195c4fe6-5eec-4971-812e-59fac0d832b1\") " pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" Jul 02 19:28:12.988092 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.987972 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wv7r\" (UniqueName: \"kubernetes.io/projected/195c4fe6-5eec-4971-812e-59fac0d832b1-kube-api-access-4wv7r\") pod \"maas-api-e2e-derive-079596c4-65cc46fbd9-z7556\" (UID: \"195c4fe6-5eec-4971-812e-59fac0d832b1\") " pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" Jul 02 19:28:12.988092 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:28:12.988085 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-079596c4-serving-cert: secret "maas-api-e2e-derive-079596c4-serving-cert" not found Jul 02 19:28:12.988264 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:28:12.988144 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195c4fe6-5eec-4971-812e-59fac0d832b1-maas-api-tls podName:195c4fe6-5eec-4971-812e-59fac0d832b1 nodeName:}" failed. No retries permitted until 2026-07-02 19:28:13.488122895 +0000 UTC m=+1523.284993065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/195c4fe6-5eec-4971-812e-59fac0d832b1-maas-api-tls") pod "maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" (UID: "195c4fe6-5eec-4971-812e-59fac0d832b1") : secret "maas-api-e2e-derive-079596c4-serving-cert" not found Jul 02 19:28:12.988580 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.988557 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c4fe6-5eec-4971-812e-59fac0d832b1-openshift-service-ca\") pod \"maas-api-e2e-derive-079596c4-65cc46fbd9-z7556\" (UID: \"195c4fe6-5eec-4971-812e-59fac0d832b1\") " pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" Jul 02 19:28:12.997361 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:12.997336 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wv7r\" (UniqueName: \"kubernetes.io/projected/195c4fe6-5eec-4971-812e-59fac0d832b1-kube-api-access-4wv7r\") pod \"maas-api-e2e-derive-079596c4-65cc46fbd9-z7556\" (UID: \"195c4fe6-5eec-4971-812e-59fac0d832b1\") " pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" Jul 02 19:28:13.320154 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:13.320120 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:13.479462 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:13.479436 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:13.492302 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:13.492271 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195c4fe6-5eec-4971-812e-59fac0d832b1-maas-api-tls\") pod \"maas-api-e2e-derive-079596c4-65cc46fbd9-z7556\" (UID: \"195c4fe6-5eec-4971-812e-59fac0d832b1\") " pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" Jul 02 19:28:13.494792 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:13.494768 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195c4fe6-5eec-4971-812e-59fac0d832b1-maas-api-tls\") pod \"maas-api-e2e-derive-079596c4-65cc46fbd9-z7556\" (UID: \"195c4fe6-5eec-4971-812e-59fac0d832b1\") " pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" Jul 02 19:28:13.587222 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:13.587142 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:13.764413 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:13.764374 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" Jul 02 19:28:13.895942 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:13.895905 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556"] Jul 02 19:28:13.898784 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:28:13.898751 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195c4fe6_5eec_4971_812e_59fac0d832b1.slice/crio-42a13d36f668c683f9499ab61d3565a53399a17781cb2cc1d8a6bcbb1403bdf7 WatchSource:0}: Error finding container 42a13d36f668c683f9499ab61d3565a53399a17781cb2cc1d8a6bcbb1403bdf7: Status 404 returned error can't find the container with id 42a13d36f668c683f9499ab61d3565a53399a17781cb2cc1d8a6bcbb1403bdf7 Jul 02 19:28:14.067416 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:14.067381 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" event={"ID":"195c4fe6-5eec-4971-812e-59fac0d832b1","Type":"ContainerStarted","Data":"42a13d36f668c683f9499ab61d3565a53399a17781cb2cc1d8a6bcbb1403bdf7"} Jul 02 19:28:14.963982 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:14.963950 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:15.072822 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:15.072789 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" event={"ID":"195c4fe6-5eec-4971-812e-59fac0d832b1","Type":"ContainerStarted","Data":"6a9402912d393ac453c4feef296df493c1abf1da7a7e08b60d0a163c06f05c32"} Jul 02 19:28:15.073035 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:15.072931 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" Jul 02 19:28:15.092627 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:15.092578 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" podStartSLOduration=2.745246565 podStartE2EDuration="3.092562091s" totalImagesPullingTime="347.315526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:28:15.091425608 +0000 UTC m=+1524.888295804" watchObservedRunningTime="2026-07-02 19:28:15.092562091 +0000 UTC m=+1524.889432282" Jul 02 19:28:17.268246 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:17.268209 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:17.446917 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:17.446885 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:17.575479 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:17.575374 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:17.678034 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:17.678005 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556"] Jul 02 19:28:17.678220 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:17.678188 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" podUID="195c4fe6-5eec-4971-812e-59fac0d832b1" containerName="maas-api" containerID="cri-o://6a9402912d393ac453c4feef296df493c1abf1da7a7e08b60d0a163c06f05c32" gracePeriod=30 Jul 02 19:28:17.708190 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:17.708164 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:17.921101 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:17.921079 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" Jul 02 19:28:18.031338 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.031300 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c4fe6-5eec-4971-812e-59fac0d832b1-openshift-service-ca\") pod \"195c4fe6-5eec-4971-812e-59fac0d832b1\" (UID: \"195c4fe6-5eec-4971-812e-59fac0d832b1\") " Jul 02 19:28:18.031338 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.031342 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wv7r\" (UniqueName: \"kubernetes.io/projected/195c4fe6-5eec-4971-812e-59fac0d832b1-kube-api-access-4wv7r\") pod \"195c4fe6-5eec-4971-812e-59fac0d832b1\" (UID: \"195c4fe6-5eec-4971-812e-59fac0d832b1\") " Jul 02 19:28:18.031555 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.031435 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195c4fe6-5eec-4971-812e-59fac0d832b1-maas-api-tls\") pod \"195c4fe6-5eec-4971-812e-59fac0d832b1\" (UID: \"195c4fe6-5eec-4971-812e-59fac0d832b1\") " Jul 02 19:28:18.031753 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.031722 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195c4fe6-5eec-4971-812e-59fac0d832b1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "195c4fe6-5eec-4971-812e-59fac0d832b1" (UID: "195c4fe6-5eec-4971-812e-59fac0d832b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:28:18.033684 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.033657 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195c4fe6-5eec-4971-812e-59fac0d832b1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "195c4fe6-5eec-4971-812e-59fac0d832b1" (UID: "195c4fe6-5eec-4971-812e-59fac0d832b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:28:18.033684 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.033676 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195c4fe6-5eec-4971-812e-59fac0d832b1-kube-api-access-4wv7r" (OuterVolumeSpecName: "kube-api-access-4wv7r") pod "195c4fe6-5eec-4971-812e-59fac0d832b1" (UID: "195c4fe6-5eec-4971-812e-59fac0d832b1"). InnerVolumeSpecName "kube-api-access-4wv7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:28:18.086408 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.086374 2565 generic.go:356] "Generic (PLEG): container finished" podID="195c4fe6-5eec-4971-812e-59fac0d832b1" containerID="6a9402912d393ac453c4feef296df493c1abf1da7a7e08b60d0a163c06f05c32" exitCode=0 Jul 02 19:28:18.086574 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.086421 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" Jul 02 19:28:18.086574 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.086425 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" event={"ID":"195c4fe6-5eec-4971-812e-59fac0d832b1","Type":"ContainerDied","Data":"6a9402912d393ac453c4feef296df493c1abf1da7a7e08b60d0a163c06f05c32"} Jul 02 19:28:18.086574 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.086527 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556" event={"ID":"195c4fe6-5eec-4971-812e-59fac0d832b1","Type":"ContainerDied","Data":"42a13d36f668c683f9499ab61d3565a53399a17781cb2cc1d8a6bcbb1403bdf7"} Jul 02 19:28:18.086574 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.086549 2565 scope.go:117] "RemoveContainer" containerID="6a9402912d393ac453c4feef296df493c1abf1da7a7e08b60d0a163c06f05c32" Jul 02 19:28:18.098650 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.098630 2565 scope.go:117] "RemoveContainer" containerID="6a9402912d393ac453c4feef296df493c1abf1da7a7e08b60d0a163c06f05c32" Jul 02 19:28:18.098940 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:28:18.098922 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a9402912d393ac453c4feef296df493c1abf1da7a7e08b60d0a163c06f05c32\": container with ID starting with 6a9402912d393ac453c4feef296df493c1abf1da7a7e08b60d0a163c06f05c32 not found: ID does not exist" containerID="6a9402912d393ac453c4feef296df493c1abf1da7a7e08b60d0a163c06f05c32" Jul 02 19:28:18.098995 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.098950 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9402912d393ac453c4feef296df493c1abf1da7a7e08b60d0a163c06f05c32"} err="failed to get container status \"6a9402912d393ac453c4feef296df493c1abf1da7a7e08b60d0a163c06f05c32\": rpc error: code = NotFound desc = could not find container \"6a9402912d393ac453c4feef296df493c1abf1da7a7e08b60d0a163c06f05c32\": container with ID starting with 6a9402912d393ac453c4feef296df493c1abf1da7a7e08b60d0a163c06f05c32 not found: ID does not exist" Jul 02 19:28:18.111543 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.111517 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556"] Jul 02 19:28:18.117284 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.117261 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-079596c4-65cc46fbd9-z7556"] Jul 02 19:28:18.132360 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.132333 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/195c4fe6-5eec-4971-812e-59fac0d832b1-maas-api-tls\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:28:18.132360 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.132362 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/195c4fe6-5eec-4971-812e-59fac0d832b1-openshift-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:28:18.132495 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.132372 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wv7r\" (UniqueName: \"kubernetes.io/projected/195c4fe6-5eec-4971-812e-59fac0d832b1-kube-api-access-4wv7r\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:28:18.763262 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:18.763232 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="195c4fe6-5eec-4971-812e-59fac0d832b1" path="/var/lib/kubelet/pods/195c4fe6-5eec-4971-812e-59fac0d832b1/volumes" Jul 02 19:28:19.404994 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:19.404967 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:19.506000 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:19.505971 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:23.493792 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:23.493756 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:23.658439 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:23.658398 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:23.997868 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:23.997816 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:24.082648 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:24.082613 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:24.181762 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:24.181732 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:25.635041 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:25.634983 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:31.899299 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:31.899258 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:32.002616 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:32.002584 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:32.096541 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:32.096508 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:32.197168 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:32.197086 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:32.624155 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:32.624124 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:28:58.712911 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:28:58.712872 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:04.536948 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:04.536908 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:04.638990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:04.638956 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:04.739530 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:04.739502 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:20.731775 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:20.731735 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:26.886879 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:26.886824 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:26.995810 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:26.995777 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:27.088049 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:27.088015 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:37.669659 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:37.669606 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:38.550927 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:38.550896 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:44.324991 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:44.324869 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:44.387748 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:44.387684 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:44.483706 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:44.483675 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:44.581720 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:44.581639 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:44.682595 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:44.682566 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:45.414329 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:45.414294 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:45.513635 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:45.513597 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:45.613896 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:45.613822 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:47.262151 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:47.262117 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:53.267042 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:53.266995 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:53.348485 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:53.348445 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:29:53.977587 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:29:53.977536 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:00.107879 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:00.107816 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:00.260018 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:00.259979 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:00.394511 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:00.394438 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:00.822114 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:00.822073 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:01.048101 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:01.048045 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:03.895352 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:03.895322 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v"] Jul 02 19:30:03.895738 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:03.895624 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195c4fe6-5eec-4971-812e-59fac0d832b1" containerName="maas-api" Jul 02 19:30:03.895738 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:03.895641 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="195c4fe6-5eec-4971-812e-59fac0d832b1" containerName="maas-api" Jul 02 19:30:03.895738 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:03.895709 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="195c4fe6-5eec-4971-812e-59fac0d832b1" containerName="maas-api" Jul 02 19:30:03.898604 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:03.898586 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:03.901056 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:03.901039 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-0d36c06a-serving-cert\"" Jul 02 19:30:03.901214 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:03.901196 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxm9f\"" Jul 02 19:30:03.908414 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:03.908392 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v"] Jul 02 19:30:04.035528 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.035494 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82x2q\" (UniqueName: \"kubernetes.io/projected/c52e8854-bf48-4f29-8bba-e6e822e4448b-kube-api-access-82x2q\") pod \"maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v\" (UID: \"c52e8854-bf48-4f29-8bba-e6e822e4448b\") " pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:04.035528 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.035533 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c52e8854-bf48-4f29-8bba-e6e822e4448b-maas-api-tls\") pod \"maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v\" (UID: \"c52e8854-bf48-4f29-8bba-e6e822e4448b\") " pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:04.035743 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.035592 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52e8854-bf48-4f29-8bba-e6e822e4448b-openshift-service-ca\") pod \"maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v\" (UID: \"c52e8854-bf48-4f29-8bba-e6e822e4448b\") " pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:04.136831 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.136791 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82x2q\" (UniqueName: \"kubernetes.io/projected/c52e8854-bf48-4f29-8bba-e6e822e4448b-kube-api-access-82x2q\") pod \"maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v\" (UID: \"c52e8854-bf48-4f29-8bba-e6e822e4448b\") " pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:04.136831 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.136830 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c52e8854-bf48-4f29-8bba-e6e822e4448b-maas-api-tls\") pod \"maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v\" (UID: \"c52e8854-bf48-4f29-8bba-e6e822e4448b\") " pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:04.137066 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.136888 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52e8854-bf48-4f29-8bba-e6e822e4448b-openshift-service-ca\") pod \"maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v\" (UID: \"c52e8854-bf48-4f29-8bba-e6e822e4448b\") " pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:04.137066 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:30:04.137002 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-0d36c06a-serving-cert: secret "maas-api-e2e-mt-0d36c06a-serving-cert" not found Jul 02 19:30:04.137135 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:30:04.137078 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c52e8854-bf48-4f29-8bba-e6e822e4448b-maas-api-tls podName:c52e8854-bf48-4f29-8bba-e6e822e4448b nodeName:}" failed. No retries permitted until 2026-07-02 19:30:04.637054668 +0000 UTC m=+1634.433924852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c52e8854-bf48-4f29-8bba-e6e822e4448b-maas-api-tls") pod "maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" (UID: "c52e8854-bf48-4f29-8bba-e6e822e4448b") : secret "maas-api-e2e-mt-0d36c06a-serving-cert" not found Jul 02 19:30:04.137454 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.137437 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52e8854-bf48-4f29-8bba-e6e822e4448b-openshift-service-ca\") pod \"maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v\" (UID: \"c52e8854-bf48-4f29-8bba-e6e822e4448b\") " pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:04.155675 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.155594 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82x2q\" (UniqueName: \"kubernetes.io/projected/c52e8854-bf48-4f29-8bba-e6e822e4448b-kube-api-access-82x2q\") pod \"maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v\" (UID: \"c52e8854-bf48-4f29-8bba-e6e822e4448b\") " pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:04.325931 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.325901 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:04.424834 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.424714 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:04.525302 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.525268 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:04.637333 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.637299 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:04.641498 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.641472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c52e8854-bf48-4f29-8bba-e6e822e4448b-maas-api-tls\") pod \"maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v\" (UID: \"c52e8854-bf48-4f29-8bba-e6e822e4448b\") " pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:04.644082 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.644061 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c52e8854-bf48-4f29-8bba-e6e822e4448b-maas-api-tls\") pod \"maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v\" (UID: \"c52e8854-bf48-4f29-8bba-e6e822e4448b\") " pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:04.811108 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.811073 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:04.943397 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:04.943363 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v"] Jul 02 19:30:04.946085 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:30:04.946054 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52e8854_bf48_4f29_8bba_e6e822e4448b.slice/crio-afa97aa9f8e5ec1f46bf4ebf13329863b52818394e26b2971aa979571e15255c WatchSource:0}: Error finding container afa97aa9f8e5ec1f46bf4ebf13329863b52818394e26b2971aa979571e15255c: Status 404 returned error can't find the container with id afa97aa9f8e5ec1f46bf4ebf13329863b52818394e26b2971aa979571e15255c Jul 02 19:30:05.537547 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:05.537514 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" event={"ID":"c52e8854-bf48-4f29-8bba-e6e822e4448b","Type":"ContainerStarted","Data":"afa97aa9f8e5ec1f46bf4ebf13329863b52818394e26b2971aa979571e15255c"} Jul 02 19:30:06.543293 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:06.543258 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" event={"ID":"c52e8854-bf48-4f29-8bba-e6e822e4448b","Type":"ContainerStarted","Data":"9bedeeb0b638d6c7b38c176732cf0a5ed796743e88fa3d0f2be8147769f38ff4"} Jul 02 19:30:06.543726 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:06.543387 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:06.562044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:06.562000 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" podStartSLOduration=2.926896873 podStartE2EDuration="3.56198745s" totalImagesPullingTime="635.090577ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:30:06.56080944 +0000 UTC m=+1636.357679633" watchObservedRunningTime="2026-07-02 19:30:06.56198745 +0000 UTC m=+1636.358857641" Jul 02 19:30:08.627479 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:08.627438 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:09.047343 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:09.047306 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:09.140326 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:09.140297 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:09.512271 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:09.512246 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:12.553161 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:12.553132 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:15.662083 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:15.662052 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:15.763140 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:15.763108 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:16.290154 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:16.290119 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:16.618467 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:16.618389 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v"] Jul 02 19:30:16.618651 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:16.618626 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" podUID="c52e8854-bf48-4f29-8bba-e6e822e4448b" containerName="maas-api" containerID="cri-o://9bedeeb0b638d6c7b38c176732cf0a5ed796743e88fa3d0f2be8147769f38ff4" gracePeriod=30 Jul 02 19:30:16.652147 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:16.652104 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:16.722278 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:16.722247 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:16.859979 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:16.859952 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:17.051863 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.051811 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52e8854-bf48-4f29-8bba-e6e822e4448b-openshift-service-ca\") pod \"c52e8854-bf48-4f29-8bba-e6e822e4448b\" (UID: \"c52e8854-bf48-4f29-8bba-e6e822e4448b\") " Jul 02 19:30:17.051863 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.051870 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82x2q\" (UniqueName: \"kubernetes.io/projected/c52e8854-bf48-4f29-8bba-e6e822e4448b-kube-api-access-82x2q\") pod \"c52e8854-bf48-4f29-8bba-e6e822e4448b\" (UID: \"c52e8854-bf48-4f29-8bba-e6e822e4448b\") " Jul 02 19:30:17.052112 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.051931 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c52e8854-bf48-4f29-8bba-e6e822e4448b-maas-api-tls\") pod \"c52e8854-bf48-4f29-8bba-e6e822e4448b\" (UID: \"c52e8854-bf48-4f29-8bba-e6e822e4448b\") " Jul 02 19:30:17.052230 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.052206 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52e8854-bf48-4f29-8bba-e6e822e4448b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c52e8854-bf48-4f29-8bba-e6e822e4448b" (UID: "c52e8854-bf48-4f29-8bba-e6e822e4448b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:30:17.054067 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.054046 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52e8854-bf48-4f29-8bba-e6e822e4448b-kube-api-access-82x2q" (OuterVolumeSpecName: "kube-api-access-82x2q") pod "c52e8854-bf48-4f29-8bba-e6e822e4448b" (UID: "c52e8854-bf48-4f29-8bba-e6e822e4448b"). InnerVolumeSpecName "kube-api-access-82x2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:30:17.054133 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.054076 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c52e8854-bf48-4f29-8bba-e6e822e4448b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c52e8854-bf48-4f29-8bba-e6e822e4448b" (UID: "c52e8854-bf48-4f29-8bba-e6e822e4448b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:30:17.152882 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.152829 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52e8854-bf48-4f29-8bba-e6e822e4448b-openshift-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:30:17.152882 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.152875 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82x2q\" (UniqueName: \"kubernetes.io/projected/c52e8854-bf48-4f29-8bba-e6e822e4448b-kube-api-access-82x2q\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:30:17.152882 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.152885 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c52e8854-bf48-4f29-8bba-e6e822e4448b-maas-api-tls\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:30:17.590752 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.590716 2565 generic.go:356] "Generic (PLEG): container finished" podID="c52e8854-bf48-4f29-8bba-e6e822e4448b" containerID="9bedeeb0b638d6c7b38c176732cf0a5ed796743e88fa3d0f2be8147769f38ff4" exitCode=0 Jul 02 19:30:17.590963 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.590762 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" Jul 02 19:30:17.590963 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.590794 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" event={"ID":"c52e8854-bf48-4f29-8bba-e6e822e4448b","Type":"ContainerDied","Data":"9bedeeb0b638d6c7b38c176732cf0a5ed796743e88fa3d0f2be8147769f38ff4"} Jul 02 19:30:17.590963 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.590860 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v" event={"ID":"c52e8854-bf48-4f29-8bba-e6e822e4448b","Type":"ContainerDied","Data":"afa97aa9f8e5ec1f46bf4ebf13329863b52818394e26b2971aa979571e15255c"} Jul 02 19:30:17.590963 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.590885 2565 scope.go:117] "RemoveContainer" containerID="9bedeeb0b638d6c7b38c176732cf0a5ed796743e88fa3d0f2be8147769f38ff4" Jul 02 19:30:17.607145 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.607128 2565 scope.go:117] "RemoveContainer" containerID="9bedeeb0b638d6c7b38c176732cf0a5ed796743e88fa3d0f2be8147769f38ff4" Jul 02 19:30:17.607391 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:30:17.607373 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bedeeb0b638d6c7b38c176732cf0a5ed796743e88fa3d0f2be8147769f38ff4\": container with ID starting with 9bedeeb0b638d6c7b38c176732cf0a5ed796743e88fa3d0f2be8147769f38ff4 not found: ID does not exist" containerID="9bedeeb0b638d6c7b38c176732cf0a5ed796743e88fa3d0f2be8147769f38ff4" Jul 02 19:30:17.607435 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.607398 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bedeeb0b638d6c7b38c176732cf0a5ed796743e88fa3d0f2be8147769f38ff4"} err="failed to get container status \"9bedeeb0b638d6c7b38c176732cf0a5ed796743e88fa3d0f2be8147769f38ff4\": rpc error: code = NotFound desc = could not find container \"9bedeeb0b638d6c7b38c176732cf0a5ed796743e88fa3d0f2be8147769f38ff4\": container with ID starting with 9bedeeb0b638d6c7b38c176732cf0a5ed796743e88fa3d0f2be8147769f38ff4 not found: ID does not exist" Jul 02 19:30:17.616345 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.616323 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v"] Jul 02 19:30:17.620405 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:17.620384 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-0d36c06a-56d84d88f9-2d76v"] Jul 02 19:30:18.763546 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:18.763511 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c52e8854-bf48-4f29-8bba-e6e822e4448b" path="/var/lib/kubelet/pods/c52e8854-bf48-4f29-8bba-e6e822e4448b/volumes" Jul 02 19:30:22.546858 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:22.546809 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:22.651744 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:22.651711 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:22.844751 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:22.844679 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:23.107465 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:23.107385 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:23.215766 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:23.215733 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:23.314902 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:23.314877 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:23.428103 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:23.428023 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:29.095918 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:29.095878 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:29.164866 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:29.164814 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:29.255617 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:29.255585 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:29.448223 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:29.448144 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:35.305318 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:35.305284 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:36.587597 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:36.587561 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:41.835089 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:41.835056 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:42.181168 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:42.181087 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:42.921291 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:42.921263 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:44.935335 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:44.935305 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:45.010458 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.010424 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4"] Jul 02 19:30:45.010735 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.010724 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52e8854-bf48-4f29-8bba-e6e822e4448b" containerName="maas-api" Jul 02 19:30:45.010777 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.010738 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52e8854-bf48-4f29-8bba-e6e822e4448b" containerName="maas-api" Jul 02 19:30:45.010807 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.010790 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52e8854-bf48-4f29-8bba-e6e822e4448b" containerName="maas-api" Jul 02 19:30:45.019315 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.019282 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:30:45.022216 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.022191 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-ca91d730-serving-cert\"" Jul 02 19:30:45.022349 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.022314 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxm9f\"" Jul 02 19:30:45.067554 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.067526 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca450cfa-4a14-42c5-b8a8-c55a21693b29-openshift-service-ca\") pod \"maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4\" (UID: \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\") " pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:30:45.067554 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.067555 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2scxp\" (UniqueName: \"kubernetes.io/projected/ca450cfa-4a14-42c5-b8a8-c55a21693b29-kube-api-access-2scxp\") pod \"maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4\" (UID: \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\") " pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:30:45.067727 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.067632 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca450cfa-4a14-42c5-b8a8-c55a21693b29-maas-api-tls\") pod \"maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4\" (UID: \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\") " pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:30:45.097000 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.096972 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4"] Jul 02 19:30:45.104911 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.104868 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:45.168949 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.168924 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca450cfa-4a14-42c5-b8a8-c55a21693b29-maas-api-tls\") pod \"maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4\" (UID: \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\") " pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:30:45.169092 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.168976 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca450cfa-4a14-42c5-b8a8-c55a21693b29-openshift-service-ca\") pod \"maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4\" (UID: \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\") " pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:30:45.169092 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:30:45.169067 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-ca91d730-serving-cert: secret "maas-api-e2e-mt-ca91d730-serving-cert" not found Jul 02 19:30:45.169166 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.169092 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2scxp\" (UniqueName: \"kubernetes.io/projected/ca450cfa-4a14-42c5-b8a8-c55a21693b29-kube-api-access-2scxp\") pod \"maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4\" (UID: \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\") " pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:30:45.169166 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:30:45.169147 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca450cfa-4a14-42c5-b8a8-c55a21693b29-maas-api-tls podName:ca450cfa-4a14-42c5-b8a8-c55a21693b29 nodeName:}" failed. No retries permitted until 2026-07-02 19:30:45.669128285 +0000 UTC m=+1675.465998455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca450cfa-4a14-42c5-b8a8-c55a21693b29-maas-api-tls") pod "maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" (UID: "ca450cfa-4a14-42c5-b8a8-c55a21693b29") : secret "maas-api-e2e-mt-ca91d730-serving-cert" not found Jul 02 19:30:45.169543 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.169527 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca450cfa-4a14-42c5-b8a8-c55a21693b29-openshift-service-ca\") pod \"maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4\" (UID: \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\") " pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:30:45.183167 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.183143 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2scxp\" (UniqueName: \"kubernetes.io/projected/ca450cfa-4a14-42c5-b8a8-c55a21693b29-kube-api-access-2scxp\") pod \"maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4\" (UID: \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\") " pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:30:45.275643 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.275600 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:45.398022 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.397995 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:45.673927 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.673818 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca450cfa-4a14-42c5-b8a8-c55a21693b29-maas-api-tls\") pod \"maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4\" (UID: \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\") " pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:30:45.676330 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.676309 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca450cfa-4a14-42c5-b8a8-c55a21693b29-maas-api-tls\") pod \"maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4\" (UID: \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\") " pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:30:45.820939 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.820904 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:45.923561 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.923529 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:45.929598 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:45.929548 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:30:46.063522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:46.063442 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4"] Jul 02 19:30:46.066099 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:30:46.066067 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca450cfa_4a14_42c5_b8a8_c55a21693b29.slice/crio-f6a0a84cc9cb7809d1fdf74d6c60eb7df6e8c0cecaef23b76b832cf664060fef WatchSource:0}: Error finding container f6a0a84cc9cb7809d1fdf74d6c60eb7df6e8c0cecaef23b76b832cf664060fef: Status 404 returned error can't find the container with id f6a0a84cc9cb7809d1fdf74d6c60eb7df6e8c0cecaef23b76b832cf664060fef Jul 02 19:30:46.720884 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:46.720836 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" event={"ID":"ca450cfa-4a14-42c5-b8a8-c55a21693b29","Type":"ContainerStarted","Data":"029876d70fcd347c067746d64e2c02f1809f9fffbd2c55c14bd8f7733675a613"} Jul 02 19:30:46.721079 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:46.720902 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" event={"ID":"ca450cfa-4a14-42c5-b8a8-c55a21693b29","Type":"ContainerStarted","Data":"f6a0a84cc9cb7809d1fdf74d6c60eb7df6e8c0cecaef23b76b832cf664060fef"} Jul 02 19:30:46.721079 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:46.720990 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:30:46.740917 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:46.740876 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" podStartSLOduration=2.309639033 podStartE2EDuration="2.740861379s" totalImagesPullingTime="431.222346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:30:46.739151039 +0000 UTC m=+1676.536021234" watchObservedRunningTime="2026-07-02 19:30:46.740861379 +0000 UTC m=+1676.537731563" Jul 02 19:30:49.412582 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:49.412549 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:30:52.730168 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:52.730135 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:30:59.841705 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:30:59.841650 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:31:11.293439 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:11.293360 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:31:11.392503 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:11.392476 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:31:18.495948 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:18.495907 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:31:22.059959 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:22.059921 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:31:22.531058 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:22.530999 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:31:22.629906 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:22.629863 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:31:22.788101 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:22.788016 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4"] Jul 02 19:31:22.788268 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:22.788223 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" podUID="ca450cfa-4a14-42c5-b8a8-c55a21693b29" containerName="maas-api" containerID="cri-o://029876d70fcd347c067746d64e2c02f1809f9fffbd2c55c14bd8f7733675a613" gracePeriod=30 Jul 02 19:31:22.817518 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:22.817495 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:31:23.029541 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.029516 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:31:23.096782 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.096717 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca450cfa-4a14-42c5-b8a8-c55a21693b29-openshift-service-ca\") pod \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\" (UID: \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\") " Jul 02 19:31:23.096782 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.096763 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2scxp\" (UniqueName: \"kubernetes.io/projected/ca450cfa-4a14-42c5-b8a8-c55a21693b29-kube-api-access-2scxp\") pod \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\" (UID: \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\") " Jul 02 19:31:23.097251 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.096834 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca450cfa-4a14-42c5-b8a8-c55a21693b29-maas-api-tls\") pod \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\" (UID: \"ca450cfa-4a14-42c5-b8a8-c55a21693b29\") " Jul 02 19:31:23.097251 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.097117 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca450cfa-4a14-42c5-b8a8-c55a21693b29-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca450cfa-4a14-42c5-b8a8-c55a21693b29" (UID: "ca450cfa-4a14-42c5-b8a8-c55a21693b29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:31:23.099018 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.098990 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca450cfa-4a14-42c5-b8a8-c55a21693b29-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca450cfa-4a14-42c5-b8a8-c55a21693b29" (UID: "ca450cfa-4a14-42c5-b8a8-c55a21693b29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:31:23.099109 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.099068 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca450cfa-4a14-42c5-b8a8-c55a21693b29-kube-api-access-2scxp" (OuterVolumeSpecName: "kube-api-access-2scxp") pod "ca450cfa-4a14-42c5-b8a8-c55a21693b29" (UID: "ca450cfa-4a14-42c5-b8a8-c55a21693b29"). InnerVolumeSpecName "kube-api-access-2scxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:31:23.197609 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.197580 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca450cfa-4a14-42c5-b8a8-c55a21693b29-openshift-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:31:23.197609 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.197604 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2scxp\" (UniqueName: \"kubernetes.io/projected/ca450cfa-4a14-42c5-b8a8-c55a21693b29-kube-api-access-2scxp\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:31:23.197609 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.197613 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca450cfa-4a14-42c5-b8a8-c55a21693b29-maas-api-tls\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:31:23.877981 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.877943 2565 generic.go:356] "Generic (PLEG): container finished" podID="ca450cfa-4a14-42c5-b8a8-c55a21693b29" containerID="029876d70fcd347c067746d64e2c02f1809f9fffbd2c55c14bd8f7733675a613" exitCode=0 Jul 02 19:31:23.878151 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.877989 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" Jul 02 19:31:23.878151 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.878036 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" event={"ID":"ca450cfa-4a14-42c5-b8a8-c55a21693b29","Type":"ContainerDied","Data":"029876d70fcd347c067746d64e2c02f1809f9fffbd2c55c14bd8f7733675a613"} Jul 02 19:31:23.878151 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.878093 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4" event={"ID":"ca450cfa-4a14-42c5-b8a8-c55a21693b29","Type":"ContainerDied","Data":"f6a0a84cc9cb7809d1fdf74d6c60eb7df6e8c0cecaef23b76b832cf664060fef"} Jul 02 19:31:23.878151 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.878120 2565 scope.go:117] "RemoveContainer" containerID="029876d70fcd347c067746d64e2c02f1809f9fffbd2c55c14bd8f7733675a613" Jul 02 19:31:23.889026 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.889010 2565 scope.go:117] "RemoveContainer" containerID="029876d70fcd347c067746d64e2c02f1809f9fffbd2c55c14bd8f7733675a613" Jul 02 19:31:23.889323 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:31:23.889295 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"029876d70fcd347c067746d64e2c02f1809f9fffbd2c55c14bd8f7733675a613\": container with ID starting with 029876d70fcd347c067746d64e2c02f1809f9fffbd2c55c14bd8f7733675a613 not found: ID does not exist" containerID="029876d70fcd347c067746d64e2c02f1809f9fffbd2c55c14bd8f7733675a613" Jul 02 19:31:23.889416 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.889336 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"029876d70fcd347c067746d64e2c02f1809f9fffbd2c55c14bd8f7733675a613"} err="failed to get container status \"029876d70fcd347c067746d64e2c02f1809f9fffbd2c55c14bd8f7733675a613\": rpc error: code = NotFound desc = could not find container \"029876d70fcd347c067746d64e2c02f1809f9fffbd2c55c14bd8f7733675a613\": container with ID starting with 029876d70fcd347c067746d64e2c02f1809f9fffbd2c55c14bd8f7733675a613 not found: ID does not exist" Jul 02 19:31:23.900718 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.900695 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4"] Jul 02 19:31:23.904287 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:23.904266 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-ca91d730-588cd6fcb6-mzjr4"] Jul 02 19:31:24.763541 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:24.763508 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca450cfa-4a14-42c5-b8a8-c55a21693b29" path="/var/lib/kubelet/pods/ca450cfa-4a14-42c5-b8a8-c55a21693b29/volumes" Jul 02 19:31:28.037667 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:28.037629 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:31:33.119290 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:31:33.119256 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:10.274621 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:10.274591 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:10.383306 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:10.383278 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:16.047815 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:16.047782 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:22.171058 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:22.171027 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:22.251779 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:22.251745 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:22.470653 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:22.470565 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:22.750423 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:22.750390 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:23.854827 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:23.854792 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:23.979970 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:23.979940 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:24.801370 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:24.801331 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:24.902489 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:24.902455 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:26.501904 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:26.501868 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:30.448362 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:30.448333 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:32.481112 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:32.481078 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:32.913265 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:32.913236 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:33.036024 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:33.035987 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:33.616712 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:33.616676 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:39.525959 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:39.525917 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:39.628219 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:39.628183 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:39.726162 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:39.726128 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:50.275287 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:50.275255 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:50.604412 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:50.604320 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:32:51.240783 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:51.240682 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:32:51.246975 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:32:51.243079 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:33:11.579127 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:11.579087 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:17.746400 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:17.746366 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:17.813182 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:17.813140 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:18.323959 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:18.323928 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:18.514011 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:18.513985 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:18.788507 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:18.788469 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:21.756466 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.756430 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:21.835177 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.835144 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:21.843524 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.843495 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74"] Jul 02 19:33:21.843831 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.843804 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca450cfa-4a14-42c5-b8a8-c55a21693b29" containerName="maas-api" Jul 02 19:33:21.843831 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.843818 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca450cfa-4a14-42c5-b8a8-c55a21693b29" containerName="maas-api" Jul 02 19:33:21.843948 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.843883 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca450cfa-4a14-42c5-b8a8-c55a21693b29" containerName="maas-api" Jul 02 19:33:21.846776 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.846760 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:33:21.849109 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.849089 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-1980a2-serving-cert\"" Jul 02 19:33:21.849513 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.849497 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxm9f\"" Jul 02 19:33:21.859372 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.859343 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74"] Jul 02 19:33:21.896542 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.896509 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/061c5c7c-88e7-448e-9531-746590b79e41-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1980a2-694984b577-z5c74\" (UID: \"061c5c7c-88e7-448e-9531-746590b79e41\") " pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:33:21.896686 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.896580 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5rl4\" (UniqueName: \"kubernetes.io/projected/061c5c7c-88e7-448e-9531-746590b79e41-kube-api-access-k5rl4\") pod \"maas-api-e2e-inf-a-1980a2-694984b577-z5c74\" (UID: \"061c5c7c-88e7-448e-9531-746590b79e41\") " pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:33:21.896686 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.896647 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061c5c7c-88e7-448e-9531-746590b79e41-maas-api-tls\") pod \"maas-api-e2e-inf-a-1980a2-694984b577-z5c74\" (UID: \"061c5c7c-88e7-448e-9531-746590b79e41\") " pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:33:21.913114 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.913085 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:21.998159 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.998125 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/061c5c7c-88e7-448e-9531-746590b79e41-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1980a2-694984b577-z5c74\" (UID: \"061c5c7c-88e7-448e-9531-746590b79e41\") " pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:33:21.998313 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.998229 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5rl4\" (UniqueName: \"kubernetes.io/projected/061c5c7c-88e7-448e-9531-746590b79e41-kube-api-access-k5rl4\") pod \"maas-api-e2e-inf-a-1980a2-694984b577-z5c74\" (UID: \"061c5c7c-88e7-448e-9531-746590b79e41\") " pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:33:21.998313 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.998264 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061c5c7c-88e7-448e-9531-746590b79e41-maas-api-tls\") pod \"maas-api-e2e-inf-a-1980a2-694984b577-z5c74\" (UID: \"061c5c7c-88e7-448e-9531-746590b79e41\") " pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:33:21.998408 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:33:21.998384 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-a-1980a2-serving-cert: secret "maas-api-e2e-inf-a-1980a2-serving-cert" not found Jul 02 19:33:21.998461 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:33:21.998450 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/061c5c7c-88e7-448e-9531-746590b79e41-maas-api-tls podName:061c5c7c-88e7-448e-9531-746590b79e41 nodeName:}" failed. No retries permitted until 2026-07-02 19:33:22.498431168 +0000 UTC m=+1832.295301337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/061c5c7c-88e7-448e-9531-746590b79e41-maas-api-tls") pod "maas-api-e2e-inf-a-1980a2-694984b577-z5c74" (UID: "061c5c7c-88e7-448e-9531-746590b79e41") : secret "maas-api-e2e-inf-a-1980a2-serving-cert" not found Jul 02 19:33:21.998812 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:21.998795 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/061c5c7c-88e7-448e-9531-746590b79e41-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1980a2-694984b577-z5c74\" (UID: \"061c5c7c-88e7-448e-9531-746590b79e41\") " pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:33:22.009568 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:22.009515 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5rl4\" (UniqueName: \"kubernetes.io/projected/061c5c7c-88e7-448e-9531-746590b79e41-kube-api-access-k5rl4\") pod \"maas-api-e2e-inf-a-1980a2-694984b577-z5c74\" (UID: \"061c5c7c-88e7-448e-9531-746590b79e41\") " pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:33:22.310100 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:22.310006 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:22.459486 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:22.459455 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:22.502600 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:22.502571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061c5c7c-88e7-448e-9531-746590b79e41-maas-api-tls\") pod \"maas-api-e2e-inf-a-1980a2-694984b577-z5c74\" (UID: \"061c5c7c-88e7-448e-9531-746590b79e41\") " pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:33:22.505342 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:22.505311 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061c5c7c-88e7-448e-9531-746590b79e41-maas-api-tls\") pod \"maas-api-e2e-inf-a-1980a2-694984b577-z5c74\" (UID: \"061c5c7c-88e7-448e-9531-746590b79e41\") " pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:33:22.560783 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:22.560706 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:22.760006 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:22.759981 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:33:22.891444 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:22.891419 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74"] Jul 02 19:33:22.893746 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:33:22.893720 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061c5c7c_88e7_448e_9531_746590b79e41.slice/crio-7ed3d5f688a982eec9a9c6014882dde64d7fadd48e54618e835e85b6e9871434 WatchSource:0}: Error finding container 7ed3d5f688a982eec9a9c6014882dde64d7fadd48e54618e835e85b6e9871434: Status 404 returned error can't find the container with id 7ed3d5f688a982eec9a9c6014882dde64d7fadd48e54618e835e85b6e9871434 Jul 02 19:33:22.895246 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:22.895223 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:33:23.376871 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:23.376818 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" event={"ID":"061c5c7c-88e7-448e-9531-746590b79e41","Type":"ContainerStarted","Data":"7ed3d5f688a982eec9a9c6014882dde64d7fadd48e54618e835e85b6e9871434"} Jul 02 19:33:24.382298 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:24.382264 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" event={"ID":"061c5c7c-88e7-448e-9531-746590b79e41","Type":"ContainerStarted","Data":"82136a821c5a8aa14e64f99117f17d04066b313350ed8350f7e9face2ad79bf4"} Jul 02 19:33:24.382670 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:24.382395 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:33:24.400132 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:24.400092 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" podStartSLOduration=2.890719768 podStartE2EDuration="3.400077423s" totalImagesPullingTime="509.357655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:33:24.398910676 +0000 UTC m=+1834.195780867" watchObservedRunningTime="2026-07-02 19:33:24.400077423 +0000 UTC m=+1834.196947613" Jul 02 19:33:26.185764 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:26.185732 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:26.510718 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:26.510691 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:26.920402 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:26.920299 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:27.025308 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:27.025276 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:29.815098 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:29.815069 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:29.908008 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:29.907969 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:29.982358 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:29.982320 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z"] Jul 02 19:33:29.985787 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:29.985769 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" Jul 02 19:33:29.988213 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:29.988187 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-85423d-serving-cert\"" Jul 02 19:33:29.996868 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:29.996828 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z"] Jul 02 19:33:30.029642 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.029612 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:30.063985 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.063939 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/718cc44b-8079-4380-a31e-4f0e33c70a4b-maas-api-tls\") pod \"maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z\" (UID: \"718cc44b-8079-4380-a31e-4f0e33c70a4b\") " pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" Jul 02 19:33:30.063985 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.063985 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4txk\" (UniqueName: \"kubernetes.io/projected/718cc44b-8079-4380-a31e-4f0e33c70a4b-kube-api-access-r4txk\") pod \"maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z\" (UID: \"718cc44b-8079-4380-a31e-4f0e33c70a4b\") " pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" Jul 02 19:33:30.064193 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.064074 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/718cc44b-8079-4380-a31e-4f0e33c70a4b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z\" (UID: \"718cc44b-8079-4380-a31e-4f0e33c70a4b\") " pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" Jul 02 19:33:30.164711 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.164619 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/718cc44b-8079-4380-a31e-4f0e33c70a4b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z\" (UID: \"718cc44b-8079-4380-a31e-4f0e33c70a4b\") " pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" Jul 02 19:33:30.164711 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.164681 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/718cc44b-8079-4380-a31e-4f0e33c70a4b-maas-api-tls\") pod \"maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z\" (UID: \"718cc44b-8079-4380-a31e-4f0e33c70a4b\") " pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" Jul 02 19:33:30.164970 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.164719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4txk\" (UniqueName: \"kubernetes.io/projected/718cc44b-8079-4380-a31e-4f0e33c70a4b-kube-api-access-r4txk\") pod \"maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z\" (UID: \"718cc44b-8079-4380-a31e-4f0e33c70a4b\") " pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" Jul 02 19:33:30.165376 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.165351 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/718cc44b-8079-4380-a31e-4f0e33c70a4b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z\" (UID: \"718cc44b-8079-4380-a31e-4f0e33c70a4b\") " pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" Jul 02 19:33:30.167297 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.167275 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/718cc44b-8079-4380-a31e-4f0e33c70a4b-maas-api-tls\") pod \"maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z\" (UID: \"718cc44b-8079-4380-a31e-4f0e33c70a4b\") " pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" Jul 02 19:33:30.173995 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.173969 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4txk\" (UniqueName: \"kubernetes.io/projected/718cc44b-8079-4380-a31e-4f0e33c70a4b-kube-api-access-r4txk\") pod \"maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z\" (UID: \"718cc44b-8079-4380-a31e-4f0e33c70a4b\") " pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" Jul 02 19:33:30.202340 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.202312 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:30.296582 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.296539 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" Jul 02 19:33:30.392702 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.392674 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:33:30.435761 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.435737 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z"] Jul 02 19:33:30.438231 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:33:30.438198 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod718cc44b_8079_4380_a31e_4f0e33c70a4b.slice/crio-5b249718e535a8b32a93ec3a1e6a588049e8aee637d71dbd481f57929d89591d WatchSource:0}: Error finding container 5b249718e535a8b32a93ec3a1e6a588049e8aee637d71dbd481f57929d89591d: Status 404 returned error can't find the container with id 5b249718e535a8b32a93ec3a1e6a588049e8aee637d71dbd481f57929d89591d Jul 02 19:33:30.485814 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.485780 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:30.592900 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:30.592826 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:31.414317 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:31.414279 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" event={"ID":"718cc44b-8079-4380-a31e-4f0e33c70a4b","Type":"ContainerStarted","Data":"2c4167814f324271674505f2e218b50277abb3a0d762771cac7c66cd6d512cee"} Jul 02 19:33:31.414708 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:31.414378 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" Jul 02 19:33:31.414708 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:31.414405 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" event={"ID":"718cc44b-8079-4380-a31e-4f0e33c70a4b","Type":"ContainerStarted","Data":"5b249718e535a8b32a93ec3a1e6a588049e8aee637d71dbd481f57929d89591d"} Jul 02 19:33:31.451035 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:31.450983 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" podStartSLOduration=1.98982993 podStartE2EDuration="2.450966346s" totalImagesPullingTime="461.136416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:33:31.448734478 +0000 UTC m=+1841.245604672" watchObservedRunningTime="2026-07-02 19:33:31.450966346 +0000 UTC m=+1841.247836537" Jul 02 19:33:34.202737 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:34.202701 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:37.423974 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:37.423947 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" Jul 02 19:33:43.207230 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:43.207202 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:43.308327 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:43.308294 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:48.054226 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:48.054194 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:52.192593 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:52.192555 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:52.292542 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:52.292509 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:55.875579 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:55.875548 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:33:55.976055 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:33:55.976028 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:24.756670 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.756639 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4"] Jul 02 19:34:24.760129 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.760107 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.769973 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.769946 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-85423d\"/\"openshift-service-ca.crt\"" Jul 02 19:34:24.770360 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.770320 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-85423d\"/\"kube-root-ca.crt\"" Jul 02 19:34:24.770453 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.770404 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-85423d\"/\"test-model-85423d-kserve-self-signed-certs\"" Jul 02 19:34:24.770534 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.770315 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-85423d\"/\"default-dockercfg-q2ztr\"" Jul 02 19:34:24.783979 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.783953 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4"] Jul 02 19:34:24.810452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.810427 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:24.824604 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.824576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sc6t\" (UniqueName: \"kubernetes.io/projected/97697ec3-0fcc-4ebb-9d86-6e2446e06700-kube-api-access-7sc6t\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.824709 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.824613 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-model-cache\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.824709 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.824676 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-home\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.824800 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.824729 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97697ec3-0fcc-4ebb-9d86-6e2446e06700-tls-certs\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.824871 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.824813 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-dshm\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.824871 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.824829 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-tmp-dir\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.925926 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.925882 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-dshm\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.926078 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.925929 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-tmp-dir\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.926078 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.925966 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sc6t\" (UniqueName: \"kubernetes.io/projected/97697ec3-0fcc-4ebb-9d86-6e2446e06700-kube-api-access-7sc6t\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.926078 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.925992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-model-cache\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.926078 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.926028 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-home\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.926285 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.926077 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97697ec3-0fcc-4ebb-9d86-6e2446e06700-tls-certs\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.926402 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.926384 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-tmp-dir\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.926486 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.926466 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-home\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.926486 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.926481 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-model-cache\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.928346 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.928315 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-dshm\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.928576 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.928560 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97697ec3-0fcc-4ebb-9d86-6e2446e06700-tls-certs\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:24.941046 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:24.941028 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sc6t\" (UniqueName: \"kubernetes.io/projected/97697ec3-0fcc-4ebb-9d86-6e2446e06700-kube-api-access-7sc6t\") pod \"test-model-85423d-kserve-8697bc9d97-8qbn4\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:25.070459 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:25.070384 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:25.202473 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:25.202444 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4"] Jul 02 19:34:25.204892 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:34:25.204830 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97697ec3_0fcc_4ebb_9d86_6e2446e06700.slice/crio-232de6b5659d61ec85faa396c87b870ae4fa039a45329e47cae1ace4ccc03bfb WatchSource:0}: Error finding container 232de6b5659d61ec85faa396c87b870ae4fa039a45329e47cae1ace4ccc03bfb: Status 404 returned error can't find the container with id 232de6b5659d61ec85faa396c87b870ae4fa039a45329e47cae1ace4ccc03bfb Jul 02 19:34:25.637576 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:25.637542 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" event={"ID":"97697ec3-0fcc-4ebb-9d86-6e2446e06700","Type":"ContainerStarted","Data":"b8a4f435dacc6ffa944d32dba99f688587651d90d7faee348afdea243c48cf17"} Jul 02 19:34:25.637576 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:25.637579 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" event={"ID":"97697ec3-0fcc-4ebb-9d86-6e2446e06700","Type":"ContainerStarted","Data":"232de6b5659d61ec85faa396c87b870ae4fa039a45329e47cae1ace4ccc03bfb"} Jul 02 19:34:25.656531 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:25.656486 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" podStartSLOduration=1.6564718630000002 podStartE2EDuration="1.656471863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:34:25.654761944 +0000 UTC m=+1895.451632154" watchObservedRunningTime="2026-07-02 19:34:25.656471863 +0000 UTC m=+1895.453342054" Jul 02 19:34:35.071136 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:35.071096 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:35.071136 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:35.071145 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:35.083427 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:35.083401 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:35.692661 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:35.692630 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:43.086955 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:43.086916 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:44.033443 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:44.033408 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4"] Jul 02 19:34:44.058564 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:44.058529 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:44.386035 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:44.385951 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:44.744663 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:44.744588 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:44.844543 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:44.844510 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:44.945350 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:44.945317 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:45.045556 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.045527 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74"] Jul 02 19:34:45.045787 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.045760 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" podUID="061c5c7c-88e7-448e-9531-746590b79e41" containerName="maas-api" containerID="cri-o://82136a821c5a8aa14e64f99117f17d04066b313350ed8350f7e9face2ad79bf4" gracePeriod=30 Jul 02 19:34:45.075936 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.075907 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:45.387881 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.387829 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" podUID="061c5c7c-88e7-448e-9531-746590b79e41" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.30:8443/health\": dial tcp 10.134.0.30:8443: connect: connection refused" Jul 02 19:34:45.490252 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.490230 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:34:45.499380 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.499351 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5rl4\" (UniqueName: \"kubernetes.io/projected/061c5c7c-88e7-448e-9531-746590b79e41-kube-api-access-k5rl4\") pod \"061c5c7c-88e7-448e-9531-746590b79e41\" (UID: \"061c5c7c-88e7-448e-9531-746590b79e41\") " Jul 02 19:34:45.499481 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.499468 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061c5c7c-88e7-448e-9531-746590b79e41-maas-api-tls\") pod \"061c5c7c-88e7-448e-9531-746590b79e41\" (UID: \"061c5c7c-88e7-448e-9531-746590b79e41\") " Jul 02 19:34:45.499531 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.499514 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/061c5c7c-88e7-448e-9531-746590b79e41-openshift-service-ca\") pod \"061c5c7c-88e7-448e-9531-746590b79e41\" (UID: \"061c5c7c-88e7-448e-9531-746590b79e41\") " Jul 02 19:34:45.499934 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.499906 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061c5c7c-88e7-448e-9531-746590b79e41-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "061c5c7c-88e7-448e-9531-746590b79e41" (UID: "061c5c7c-88e7-448e-9531-746590b79e41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:34:45.501669 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.501628 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061c5c7c-88e7-448e-9531-746590b79e41-kube-api-access-k5rl4" (OuterVolumeSpecName: "kube-api-access-k5rl4") pod "061c5c7c-88e7-448e-9531-746590b79e41" (UID: "061c5c7c-88e7-448e-9531-746590b79e41"). InnerVolumeSpecName "kube-api-access-k5rl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:34:45.501815 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.501789 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/061c5c7c-88e7-448e-9531-746590b79e41-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "061c5c7c-88e7-448e-9531-746590b79e41" (UID: "061c5c7c-88e7-448e-9531-746590b79e41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:34:45.601092 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.601008 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/061c5c7c-88e7-448e-9531-746590b79e41-maas-api-tls\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:34:45.601092 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.601047 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/061c5c7c-88e7-448e-9531-746590b79e41-openshift-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:34:45.601092 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.601057 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5rl4\" (UniqueName: \"kubernetes.io/projected/061c5c7c-88e7-448e-9531-746590b79e41-kube-api-access-k5rl4\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:34:45.724425 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.724390 2565 generic.go:356] "Generic (PLEG): container finished" podID="061c5c7c-88e7-448e-9531-746590b79e41" containerID="82136a821c5a8aa14e64f99117f17d04066b313350ed8350f7e9face2ad79bf4" exitCode=0 Jul 02 19:34:45.724558 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.724434 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" Jul 02 19:34:45.724558 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.724442 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" event={"ID":"061c5c7c-88e7-448e-9531-746590b79e41","Type":"ContainerDied","Data":"82136a821c5a8aa14e64f99117f17d04066b313350ed8350f7e9face2ad79bf4"} Jul 02 19:34:45.724558 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.724484 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74" event={"ID":"061c5c7c-88e7-448e-9531-746590b79e41","Type":"ContainerDied","Data":"7ed3d5f688a982eec9a9c6014882dde64d7fadd48e54618e835e85b6e9871434"} Jul 02 19:34:45.724558 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.724505 2565 scope.go:117] "RemoveContainer" containerID="82136a821c5a8aa14e64f99117f17d04066b313350ed8350f7e9face2ad79bf4" Jul 02 19:34:45.733155 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.733137 2565 scope.go:117] "RemoveContainer" containerID="82136a821c5a8aa14e64f99117f17d04066b313350ed8350f7e9face2ad79bf4" Jul 02 19:34:45.733379 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:34:45.733364 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82136a821c5a8aa14e64f99117f17d04066b313350ed8350f7e9face2ad79bf4\": container with ID starting with 82136a821c5a8aa14e64f99117f17d04066b313350ed8350f7e9face2ad79bf4 not found: ID does not exist" containerID="82136a821c5a8aa14e64f99117f17d04066b313350ed8350f7e9face2ad79bf4" Jul 02 19:34:45.733419 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.733386 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82136a821c5a8aa14e64f99117f17d04066b313350ed8350f7e9face2ad79bf4"} err="failed to get container status \"82136a821c5a8aa14e64f99117f17d04066b313350ed8350f7e9face2ad79bf4\": rpc error: code = NotFound desc = could not find container \"82136a821c5a8aa14e64f99117f17d04066b313350ed8350f7e9face2ad79bf4\": container with ID starting with 82136a821c5a8aa14e64f99117f17d04066b313350ed8350f7e9face2ad79bf4 not found: ID does not exist" Jul 02 19:34:45.744615 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.744591 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74"] Jul 02 19:34:45.747037 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:45.747018 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-1980a2-694984b577-z5c74"] Jul 02 19:34:46.762438 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:46.762397 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="061c5c7c-88e7-448e-9531-746590b79e41" path="/var/lib/kubelet/pods/061c5c7c-88e7-448e-9531-746590b79e41/volumes" Jul 02 19:34:49.818460 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:49.818425 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:34:59.054251 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.054212 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" podUID="97697ec3-0fcc-4ebb-9d86-6e2446e06700" containerName="main" containerID="cri-o://b8a4f435dacc6ffa944d32dba99f688587651d90d7faee348afdea243c48cf17" gracePeriod=45 Jul 02 19:34:59.306558 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.306508 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:59.415406 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.415378 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-dshm\") pod \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " Jul 02 19:34:59.415573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.415442 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sc6t\" (UniqueName: \"kubernetes.io/projected/97697ec3-0fcc-4ebb-9d86-6e2446e06700-kube-api-access-7sc6t\") pod \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " Jul 02 19:34:59.415573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.415475 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-model-cache\") pod \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " Jul 02 19:34:59.415573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.415500 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97697ec3-0fcc-4ebb-9d86-6e2446e06700-tls-certs\") pod \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " Jul 02 19:34:59.415573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.415538 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-tmp-dir\") pod \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " Jul 02 19:34:59.415573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.415564 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-home\") pod \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\" (UID: \"97697ec3-0fcc-4ebb-9d86-6e2446e06700\") " Jul 02 19:34:59.415892 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.415791 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-model-cache" (OuterVolumeSpecName: "model-cache") pod "97697ec3-0fcc-4ebb-9d86-6e2446e06700" (UID: "97697ec3-0fcc-4ebb-9d86-6e2446e06700"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:34:59.415892 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.415780 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97697ec3-0fcc-4ebb-9d86-6e2446e06700" (UID: "97697ec3-0fcc-4ebb-9d86-6e2446e06700"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:34:59.415892 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.415873 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-home" (OuterVolumeSpecName: "home") pod "97697ec3-0fcc-4ebb-9d86-6e2446e06700" (UID: "97697ec3-0fcc-4ebb-9d86-6e2446e06700"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:34:59.416064 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.415915 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-model-cache\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:34:59.416064 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.415928 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-tmp-dir\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:34:59.417961 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.417912 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97697ec3-0fcc-4ebb-9d86-6e2446e06700-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97697ec3-0fcc-4ebb-9d86-6e2446e06700" (UID: "97697ec3-0fcc-4ebb-9d86-6e2446e06700"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:34:59.418094 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.418023 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-dshm" (OuterVolumeSpecName: "dshm") pod "97697ec3-0fcc-4ebb-9d86-6e2446e06700" (UID: "97697ec3-0fcc-4ebb-9d86-6e2446e06700"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:34:59.418094 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.418029 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97697ec3-0fcc-4ebb-9d86-6e2446e06700-kube-api-access-7sc6t" (OuterVolumeSpecName: "kube-api-access-7sc6t") pod "97697ec3-0fcc-4ebb-9d86-6e2446e06700" (UID: "97697ec3-0fcc-4ebb-9d86-6e2446e06700"). InnerVolumeSpecName "kube-api-access-7sc6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:34:59.516711 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.516683 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-dshm\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:34:59.516824 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.516716 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sc6t\" (UniqueName: \"kubernetes.io/projected/97697ec3-0fcc-4ebb-9d86-6e2446e06700-kube-api-access-7sc6t\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:34:59.516824 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.516733 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97697ec3-0fcc-4ebb-9d86-6e2446e06700-tls-certs\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:34:59.516824 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.516749 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97697ec3-0fcc-4ebb-9d86-6e2446e06700-home\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:34:59.788675 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.788643 2565 generic.go:356] "Generic (PLEG): container finished" podID="97697ec3-0fcc-4ebb-9d86-6e2446e06700" containerID="b8a4f435dacc6ffa944d32dba99f688587651d90d7faee348afdea243c48cf17" exitCode=0 Jul 02 19:34:59.788675 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.788676 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" event={"ID":"97697ec3-0fcc-4ebb-9d86-6e2446e06700","Type":"ContainerDied","Data":"b8a4f435dacc6ffa944d32dba99f688587651d90d7faee348afdea243c48cf17"} Jul 02 19:34:59.788943 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.788695 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" Jul 02 19:34:59.788943 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.788711 2565 scope.go:117] "RemoveContainer" containerID="b8a4f435dacc6ffa944d32dba99f688587651d90d7faee348afdea243c48cf17" Jul 02 19:34:59.788943 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.788701 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4" event={"ID":"97697ec3-0fcc-4ebb-9d86-6e2446e06700","Type":"ContainerDied","Data":"232de6b5659d61ec85faa396c87b870ae4fa039a45329e47cae1ace4ccc03bfb"} Jul 02 19:34:59.796978 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.796962 2565 scope.go:117] "RemoveContainer" containerID="b8a4f435dacc6ffa944d32dba99f688587651d90d7faee348afdea243c48cf17" Jul 02 19:34:59.797219 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:34:59.797201 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a4f435dacc6ffa944d32dba99f688587651d90d7faee348afdea243c48cf17\": container with ID starting with b8a4f435dacc6ffa944d32dba99f688587651d90d7faee348afdea243c48cf17 not found: ID does not exist" containerID="b8a4f435dacc6ffa944d32dba99f688587651d90d7faee348afdea243c48cf17" Jul 02 19:34:59.797272 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.797225 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a4f435dacc6ffa944d32dba99f688587651d90d7faee348afdea243c48cf17"} err="failed to get container status \"b8a4f435dacc6ffa944d32dba99f688587651d90d7faee348afdea243c48cf17\": rpc error: code = NotFound desc = could not find container \"b8a4f435dacc6ffa944d32dba99f688587651d90d7faee348afdea243c48cf17\": container with ID starting with b8a4f435dacc6ffa944d32dba99f688587651d90d7faee348afdea243c48cf17 not found: ID does not exist" Jul 02 19:34:59.809429 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.809406 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4"] Jul 02 19:34:59.813297 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:34:59.813278 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-85423d/test-model-85423d-kserve-8697bc9d97-8qbn4"] Jul 02 19:35:00.763024 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:00.762987 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97697ec3-0fcc-4ebb-9d86-6e2446e06700" path="/var/lib/kubelet/pods/97697ec3-0fcc-4ebb-9d86-6e2446e06700/volumes" Jul 02 19:35:26.723013 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:26.722982 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:27.023039 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:27.023010 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:27.629676 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:27.629647 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:27.733601 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:27.733558 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:27.859431 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:27.859387 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:27.955801 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:27.955705 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:28.126229 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.126196 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:28.180816 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.180785 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z"] Jul 02 19:35:28.181119 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.181090 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" podUID="718cc44b-8079-4380-a31e-4f0e33c70a4b" containerName="maas-api" containerID="cri-o://2c4167814f324271674505f2e218b50277abb3a0d762771cac7c66cd6d512cee" gracePeriod=30 Jul 02 19:35:28.206758 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.206677 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:28.421525 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.421502 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" Jul 02 19:35:28.476892 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.476787 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/718cc44b-8079-4380-a31e-4f0e33c70a4b-maas-api-tls\") pod \"718cc44b-8079-4380-a31e-4f0e33c70a4b\" (UID: \"718cc44b-8079-4380-a31e-4f0e33c70a4b\") " Jul 02 19:35:28.476892 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.476827 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/718cc44b-8079-4380-a31e-4f0e33c70a4b-openshift-service-ca\") pod \"718cc44b-8079-4380-a31e-4f0e33c70a4b\" (UID: \"718cc44b-8079-4380-a31e-4f0e33c70a4b\") " Jul 02 19:35:28.476892 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.476880 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4txk\" (UniqueName: \"kubernetes.io/projected/718cc44b-8079-4380-a31e-4f0e33c70a4b-kube-api-access-r4txk\") pod \"718cc44b-8079-4380-a31e-4f0e33c70a4b\" (UID: \"718cc44b-8079-4380-a31e-4f0e33c70a4b\") " Jul 02 19:35:28.477231 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.477205 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718cc44b-8079-4380-a31e-4f0e33c70a4b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "718cc44b-8079-4380-a31e-4f0e33c70a4b" (UID: "718cc44b-8079-4380-a31e-4f0e33c70a4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:35:28.479195 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.479159 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/718cc44b-8079-4380-a31e-4f0e33c70a4b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "718cc44b-8079-4380-a31e-4f0e33c70a4b" (UID: "718cc44b-8079-4380-a31e-4f0e33c70a4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:35:28.479195 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.479184 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718cc44b-8079-4380-a31e-4f0e33c70a4b-kube-api-access-r4txk" (OuterVolumeSpecName: "kube-api-access-r4txk") pod "718cc44b-8079-4380-a31e-4f0e33c70a4b" (UID: "718cc44b-8079-4380-a31e-4f0e33c70a4b"). InnerVolumeSpecName "kube-api-access-r4txk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:35:28.578091 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.578067 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/718cc44b-8079-4380-a31e-4f0e33c70a4b-maas-api-tls\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:35:28.578091 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.578091 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/718cc44b-8079-4380-a31e-4f0e33c70a4b-openshift-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:35:28.578265 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.578102 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4txk\" (UniqueName: \"kubernetes.io/projected/718cc44b-8079-4380-a31e-4f0e33c70a4b-kube-api-access-r4txk\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:35:28.910241 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.910203 2565 generic.go:356] "Generic (PLEG): container finished" podID="718cc44b-8079-4380-a31e-4f0e33c70a4b" containerID="2c4167814f324271674505f2e218b50277abb3a0d762771cac7c66cd6d512cee" exitCode=0 Jul 02 19:35:28.910650 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.910245 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" Jul 02 19:35:28.910650 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.910273 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" event={"ID":"718cc44b-8079-4380-a31e-4f0e33c70a4b","Type":"ContainerDied","Data":"2c4167814f324271674505f2e218b50277abb3a0d762771cac7c66cd6d512cee"} Jul 02 19:35:28.910650 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.910318 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z" event={"ID":"718cc44b-8079-4380-a31e-4f0e33c70a4b","Type":"ContainerDied","Data":"5b249718e535a8b32a93ec3a1e6a588049e8aee637d71dbd481f57929d89591d"} Jul 02 19:35:28.910650 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.910340 2565 scope.go:117] "RemoveContainer" containerID="2c4167814f324271674505f2e218b50277abb3a0d762771cac7c66cd6d512cee" Jul 02 19:35:28.918403 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.918383 2565 scope.go:117] "RemoveContainer" containerID="2c4167814f324271674505f2e218b50277abb3a0d762771cac7c66cd6d512cee" Jul 02 19:35:28.918629 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:35:28.918613 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c4167814f324271674505f2e218b50277abb3a0d762771cac7c66cd6d512cee\": container with ID starting with 2c4167814f324271674505f2e218b50277abb3a0d762771cac7c66cd6d512cee not found: ID does not exist" containerID="2c4167814f324271674505f2e218b50277abb3a0d762771cac7c66cd6d512cee" Jul 02 19:35:28.918668 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.918637 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c4167814f324271674505f2e218b50277abb3a0d762771cac7c66cd6d512cee"} err="failed to get container status \"2c4167814f324271674505f2e218b50277abb3a0d762771cac7c66cd6d512cee\": rpc error: code = NotFound desc = could not find container \"2c4167814f324271674505f2e218b50277abb3a0d762771cac7c66cd6d512cee\": container with ID starting with 2c4167814f324271674505f2e218b50277abb3a0d762771cac7c66cd6d512cee not found: ID does not exist" Jul 02 19:35:28.935422 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.935399 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z"] Jul 02 19:35:28.940479 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:28.940459 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-85423d-548d8d488f-x7m9z"] Jul 02 19:35:30.763341 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:30.763307 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="718cc44b-8079-4380-a31e-4f0e33c70a4b" path="/var/lib/kubelet/pods/718cc44b-8079-4380-a31e-4f0e33c70a4b/volumes" Jul 02 19:35:34.066227 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:34.066194 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:34.161831 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:34.161800 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:34.322029 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:34.321953 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:34.562656 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:34.562607 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:34.891692 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:34.891656 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:35.209012 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:35.208933 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:35.363689 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:35.363644 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:38.398189 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.398138 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:38.460829 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.460794 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg"] Jul 02 19:35:38.461234 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.461215 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="718cc44b-8079-4380-a31e-4f0e33c70a4b" containerName="maas-api" Jul 02 19:35:38.461234 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.461233 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="718cc44b-8079-4380-a31e-4f0e33c70a4b" containerName="maas-api" Jul 02 19:35:38.461387 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.461245 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="061c5c7c-88e7-448e-9531-746590b79e41" containerName="maas-api" Jul 02 19:35:38.461387 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.461250 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="061c5c7c-88e7-448e-9531-746590b79e41" containerName="maas-api" Jul 02 19:35:38.461387 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.461266 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97697ec3-0fcc-4ebb-9d86-6e2446e06700" containerName="main" Jul 02 19:35:38.461387 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.461272 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97697ec3-0fcc-4ebb-9d86-6e2446e06700" containerName="main" Jul 02 19:35:38.461387 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.461322 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="718cc44b-8079-4380-a31e-4f0e33c70a4b" containerName="maas-api" Jul 02 19:35:38.461387 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.461328 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="97697ec3-0fcc-4ebb-9d86-6e2446e06700" containerName="main" Jul 02 19:35:38.461387 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.461337 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="061c5c7c-88e7-448e-9531-746590b79e41" containerName="maas-api" Jul 02 19:35:38.465602 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.465580 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" Jul 02 19:35:38.469651 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.469628 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxm9f\"" Jul 02 19:35:38.470134 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.470115 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-55a636-serving-cert\"" Jul 02 19:35:38.478521 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.478495 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg"] Jul 02 19:35:38.555506 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.555471 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbrnd\" (UniqueName: \"kubernetes.io/projected/25d2b826-0c31-4bd2-a2d4-17a095c2443e-kube-api-access-hbrnd\") pod \"maas-api-e2e-api-a-55a636-77746d9b89-qqpbg\" (UID: \"25d2b826-0c31-4bd2-a2d4-17a095c2443e\") " pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" Jul 02 19:35:38.555695 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.555518 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25d2b826-0c31-4bd2-a2d4-17a095c2443e-maas-api-tls\") pod \"maas-api-e2e-api-a-55a636-77746d9b89-qqpbg\" (UID: \"25d2b826-0c31-4bd2-a2d4-17a095c2443e\") " pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" Jul 02 19:35:38.555695 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.555566 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d2b826-0c31-4bd2-a2d4-17a095c2443e-openshift-service-ca\") pod \"maas-api-e2e-api-a-55a636-77746d9b89-qqpbg\" (UID: \"25d2b826-0c31-4bd2-a2d4-17a095c2443e\") " pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" Jul 02 19:35:38.656364 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.656279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25d2b826-0c31-4bd2-a2d4-17a095c2443e-maas-api-tls\") pod \"maas-api-e2e-api-a-55a636-77746d9b89-qqpbg\" (UID: \"25d2b826-0c31-4bd2-a2d4-17a095c2443e\") " pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" Jul 02 19:35:38.656364 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.656335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d2b826-0c31-4bd2-a2d4-17a095c2443e-openshift-service-ca\") pod \"maas-api-e2e-api-a-55a636-77746d9b89-qqpbg\" (UID: \"25d2b826-0c31-4bd2-a2d4-17a095c2443e\") " pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" Jul 02 19:35:38.656569 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.656423 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbrnd\" (UniqueName: \"kubernetes.io/projected/25d2b826-0c31-4bd2-a2d4-17a095c2443e-kube-api-access-hbrnd\") pod \"maas-api-e2e-api-a-55a636-77746d9b89-qqpbg\" (UID: \"25d2b826-0c31-4bd2-a2d4-17a095c2443e\") " pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" Jul 02 19:35:38.657074 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.657047 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d2b826-0c31-4bd2-a2d4-17a095c2443e-openshift-service-ca\") pod \"maas-api-e2e-api-a-55a636-77746d9b89-qqpbg\" (UID: \"25d2b826-0c31-4bd2-a2d4-17a095c2443e\") " pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" Jul 02 19:35:38.658804 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.658775 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25d2b826-0c31-4bd2-a2d4-17a095c2443e-maas-api-tls\") pod \"maas-api-e2e-api-a-55a636-77746d9b89-qqpbg\" (UID: \"25d2b826-0c31-4bd2-a2d4-17a095c2443e\") " pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" Jul 02 19:35:38.680158 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.680128 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbrnd\" (UniqueName: \"kubernetes.io/projected/25d2b826-0c31-4bd2-a2d4-17a095c2443e-kube-api-access-hbrnd\") pod \"maas-api-e2e-api-a-55a636-77746d9b89-qqpbg\" (UID: \"25d2b826-0c31-4bd2-a2d4-17a095c2443e\") " pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" Jul 02 19:35:38.776774 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.776743 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" Jul 02 19:35:38.915486 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.915404 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg"] Jul 02 19:35:38.918559 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:35:38.918514 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d2b826_0c31_4bd2_a2d4_17a095c2443e.slice/crio-53953c76227f95a8fc8cbe64e3e3c4ec3361ea4acd680793e7b753f7f37ac4af WatchSource:0}: Error finding container 53953c76227f95a8fc8cbe64e3e3c4ec3361ea4acd680793e7b753f7f37ac4af: Status 404 returned error can't find the container with id 53953c76227f95a8fc8cbe64e3e3c4ec3361ea4acd680793e7b753f7f37ac4af Jul 02 19:35:38.954482 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:38.954451 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" event={"ID":"25d2b826-0c31-4bd2-a2d4-17a095c2443e","Type":"ContainerStarted","Data":"53953c76227f95a8fc8cbe64e3e3c4ec3361ea4acd680793e7b753f7f37ac4af"} Jul 02 19:35:39.654649 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:39.654616 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:39.959715 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:39.959681 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" event={"ID":"25d2b826-0c31-4bd2-a2d4-17a095c2443e","Type":"ContainerStarted","Data":"073d66fa2c1b2b349e7a987f2d283fcf7c03b1e019f56361f1217fafff748f6e"} Jul 02 19:35:39.959928 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:39.959754 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" Jul 02 19:35:39.979894 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:39.979833 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" podStartSLOduration=1.458400604 podStartE2EDuration="1.979814747s" totalImagesPullingTime="521.414143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:35:39.978770578 +0000 UTC m=+1969.775640769" watchObservedRunningTime="2026-07-02 19:35:39.979814747 +0000 UTC m=+1969.776684940" Jul 02 19:35:42.871158 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:42.871074 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:43.329736 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:43.329702 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:43.632078 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:43.632001 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:44.065221 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:44.065184 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:45.969195 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:45.969168 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" Jul 02 19:35:46.513263 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:46.513221 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:46.605220 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:46.605191 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:46.693738 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:46.693704 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6"] Jul 02 19:35:46.699827 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:46.699801 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:35:46.702204 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:46.702175 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-a3b3e6-serving-cert\"" Jul 02 19:35:46.707916 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:46.707709 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6"] Jul 02 19:35:46.740110 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:46.740078 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:46.827162 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:46.827064 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a03c85-dca8-4d37-9d87-d6b035ac6dec-openshift-service-ca\") pod \"maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6\" (UID: \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\") " pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:35:46.827162 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:46.827113 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr9tw\" (UniqueName: \"kubernetes.io/projected/42a03c85-dca8-4d37-9d87-d6b035ac6dec-kube-api-access-xr9tw\") pod \"maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6\" (UID: \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\") " pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:35:46.827354 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:46.827169 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a03c85-dca8-4d37-9d87-d6b035ac6dec-maas-api-tls\") pod \"maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6\" (UID: \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\") " pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:35:46.927820 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:46.927782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a03c85-dca8-4d37-9d87-d6b035ac6dec-openshift-service-ca\") pod \"maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6\" (UID: \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\") " pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:35:46.927820 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:46.927829 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr9tw\" (UniqueName: \"kubernetes.io/projected/42a03c85-dca8-4d37-9d87-d6b035ac6dec-kube-api-access-xr9tw\") pod \"maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6\" (UID: \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\") " pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:35:46.928095 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:46.927910 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a03c85-dca8-4d37-9d87-d6b035ac6dec-maas-api-tls\") pod \"maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6\" (UID: \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\") " pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:35:46.928095 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:35:46.927998 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-a3b3e6-serving-cert: secret "maas-api-e2e-api-b-a3b3e6-serving-cert" not found Jul 02 19:35:46.928095 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:35:46.928060 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42a03c85-dca8-4d37-9d87-d6b035ac6dec-maas-api-tls podName:42a03c85-dca8-4d37-9d87-d6b035ac6dec nodeName:}" failed. No retries permitted until 2026-07-02 19:35:47.428044556 +0000 UTC m=+1977.224914725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42a03c85-dca8-4d37-9d87-d6b035ac6dec-maas-api-tls") pod "maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" (UID: "42a03c85-dca8-4d37-9d87-d6b035ac6dec") : secret "maas-api-e2e-api-b-a3b3e6-serving-cert" not found Jul 02 19:35:46.928507 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:46.928480 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a03c85-dca8-4d37-9d87-d6b035ac6dec-openshift-service-ca\") pod \"maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6\" (UID: \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\") " pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:35:46.939447 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:46.939416 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr9tw\" (UniqueName: \"kubernetes.io/projected/42a03c85-dca8-4d37-9d87-d6b035ac6dec-kube-api-access-xr9tw\") pod \"maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6\" (UID: \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\") " pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:35:47.128283 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:47.128199 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:47.297346 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:47.297308 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:47.400571 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:47.400476 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:47.431948 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:47.431922 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a03c85-dca8-4d37-9d87-d6b035ac6dec-maas-api-tls\") pod \"maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6\" (UID: \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\") " pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:35:47.434392 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:47.434363 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a03c85-dca8-4d37-9d87-d6b035ac6dec-maas-api-tls\") pod \"maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6\" (UID: \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\") " pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:35:47.611822 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:47.611785 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:35:47.758320 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:35:47.758286 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a03c85_dca8_4d37_9d87_d6b035ac6dec.slice/crio-eb7d0be1d78aef80e47cdd6f890949f8b95952b96e53b969b8134969a0a111e1 WatchSource:0}: Error finding container eb7d0be1d78aef80e47cdd6f890949f8b95952b96e53b969b8134969a0a111e1: Status 404 returned error can't find the container with id eb7d0be1d78aef80e47cdd6f890949f8b95952b96e53b969b8134969a0a111e1 Jul 02 19:35:47.762240 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:47.762214 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6"] Jul 02 19:35:47.993287 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:47.993205 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" event={"ID":"42a03c85-dca8-4d37-9d87-d6b035ac6dec","Type":"ContainerStarted","Data":"eb7d0be1d78aef80e47cdd6f890949f8b95952b96e53b969b8134969a0a111e1"} Jul 02 19:35:48.999321 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:48.999287 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" event={"ID":"42a03c85-dca8-4d37-9d87-d6b035ac6dec","Type":"ContainerStarted","Data":"090468b3eb2a9ec6ff2bdeb408678a7170a231db37407d0d232d40ea42938761"} Jul 02 19:35:48.999706 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:48.999413 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:35:49.019755 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:49.019710 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" podStartSLOduration=2.647946467 podStartE2EDuration="3.01969644s" totalImagesPullingTime="371.749973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:35:49.01842636 +0000 UTC m=+1978.815296551" watchObservedRunningTime="2026-07-02 19:35:49.01969644 +0000 UTC m=+1978.816566630" Jul 02 19:35:51.008743 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:51.008715 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:55.009879 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:55.009830 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:35:59.127652 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:59.127615 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:59.521382 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:59.521350 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:59.652414 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:59.652383 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:59.752629 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:59.752598 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:35:59.848775 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:59.848695 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg"] Jul 02 19:35:59.848974 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:59.848952 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" podUID="25d2b826-0c31-4bd2-a2d4-17a095c2443e" containerName="maas-api" containerID="cri-o://073d66fa2c1b2b349e7a987f2d283fcf7c03b1e019f56361f1217fafff748f6e" gracePeriod=30 Jul 02 19:35:59.873234 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:35:59.873210 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:00.046270 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:00.046239 2565 generic.go:356] "Generic (PLEG): container finished" podID="25d2b826-0c31-4bd2-a2d4-17a095c2443e" containerID="073d66fa2c1b2b349e7a987f2d283fcf7c03b1e019f56361f1217fafff748f6e" exitCode=0 Jul 02 19:36:00.046437 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:00.046307 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" event={"ID":"25d2b826-0c31-4bd2-a2d4-17a095c2443e","Type":"ContainerDied","Data":"073d66fa2c1b2b349e7a987f2d283fcf7c03b1e019f56361f1217fafff748f6e"} Jul 02 19:36:00.093206 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:00.093183 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" Jul 02 19:36:00.239900 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:00.239795 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbrnd\" (UniqueName: \"kubernetes.io/projected/25d2b826-0c31-4bd2-a2d4-17a095c2443e-kube-api-access-hbrnd\") pod \"25d2b826-0c31-4bd2-a2d4-17a095c2443e\" (UID: \"25d2b826-0c31-4bd2-a2d4-17a095c2443e\") " Jul 02 19:36:00.239900 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:00.239834 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25d2b826-0c31-4bd2-a2d4-17a095c2443e-maas-api-tls\") pod \"25d2b826-0c31-4bd2-a2d4-17a095c2443e\" (UID: \"25d2b826-0c31-4bd2-a2d4-17a095c2443e\") " Jul 02 19:36:00.240274 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:00.239920 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d2b826-0c31-4bd2-a2d4-17a095c2443e-openshift-service-ca\") pod \"25d2b826-0c31-4bd2-a2d4-17a095c2443e\" (UID: \"25d2b826-0c31-4bd2-a2d4-17a095c2443e\") " Jul 02 19:36:00.240315 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:00.240275 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d2b826-0c31-4bd2-a2d4-17a095c2443e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "25d2b826-0c31-4bd2-a2d4-17a095c2443e" (UID: "25d2b826-0c31-4bd2-a2d4-17a095c2443e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:36:00.242015 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:00.241982 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25d2b826-0c31-4bd2-a2d4-17a095c2443e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "25d2b826-0c31-4bd2-a2d4-17a095c2443e" (UID: "25d2b826-0c31-4bd2-a2d4-17a095c2443e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:36:00.242129 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:00.242037 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d2b826-0c31-4bd2-a2d4-17a095c2443e-kube-api-access-hbrnd" (OuterVolumeSpecName: "kube-api-access-hbrnd") pod "25d2b826-0c31-4bd2-a2d4-17a095c2443e" (UID: "25d2b826-0c31-4bd2-a2d4-17a095c2443e"). InnerVolumeSpecName "kube-api-access-hbrnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:36:00.340951 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:00.340920 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d2b826-0c31-4bd2-a2d4-17a095c2443e-openshift-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:36:00.341086 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:00.340946 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbrnd\" (UniqueName: \"kubernetes.io/projected/25d2b826-0c31-4bd2-a2d4-17a095c2443e-kube-api-access-hbrnd\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:36:00.341086 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:00.340969 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25d2b826-0c31-4bd2-a2d4-17a095c2443e-maas-api-tls\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:36:01.051802 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:01.051731 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" Jul 02 19:36:01.051956 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:01.051734 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg" event={"ID":"25d2b826-0c31-4bd2-a2d4-17a095c2443e","Type":"ContainerDied","Data":"53953c76227f95a8fc8cbe64e3e3c4ec3361ea4acd680793e7b753f7f37ac4af"} Jul 02 19:36:01.051956 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:01.051876 2565 scope.go:117] "RemoveContainer" containerID="073d66fa2c1b2b349e7a987f2d283fcf7c03b1e019f56361f1217fafff748f6e" Jul 02 19:36:01.095064 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:01.095036 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg"] Jul 02 19:36:01.110871 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:01.110825 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-55a636-77746d9b89-qqpbg"] Jul 02 19:36:02.762663 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:02.762630 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25d2b826-0c31-4bd2-a2d4-17a095c2443e" path="/var/lib/kubelet/pods/25d2b826-0c31-4bd2-a2d4-17a095c2443e/volumes" Jul 02 19:36:05.530285 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:05.530248 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:05.630572 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:05.630534 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:05.793538 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:05.793457 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:06.032435 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:06.032399 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:06.200772 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:06.200682 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:06.620160 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:06.620127 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:06.720127 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:06.720089 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:06.821258 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:06.821225 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:06.940855 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:06.940764 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6"] Jul 02 19:36:06.941010 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:06.940981 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" podUID="42a03c85-dca8-4d37-9d87-d6b035ac6dec" containerName="maas-api" containerID="cri-o://090468b3eb2a9ec6ff2bdeb408678a7170a231db37407d0d232d40ea42938761" gracePeriod=30 Jul 02 19:36:06.977854 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:06.977813 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:07.079184 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:07.079150 2565 generic.go:356] "Generic (PLEG): container finished" podID="42a03c85-dca8-4d37-9d87-d6b035ac6dec" containerID="090468b3eb2a9ec6ff2bdeb408678a7170a231db37407d0d232d40ea42938761" exitCode=0 Jul 02 19:36:07.079303 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:07.079203 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" event={"ID":"42a03c85-dca8-4d37-9d87-d6b035ac6dec","Type":"ContainerDied","Data":"090468b3eb2a9ec6ff2bdeb408678a7170a231db37407d0d232d40ea42938761"} Jul 02 19:36:07.187623 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:07.187600 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:36:07.299768 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:07.299742 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr9tw\" (UniqueName: \"kubernetes.io/projected/42a03c85-dca8-4d37-9d87-d6b035ac6dec-kube-api-access-xr9tw\") pod \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\" (UID: \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\") " Jul 02 19:36:07.299946 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:07.299777 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a03c85-dca8-4d37-9d87-d6b035ac6dec-openshift-service-ca\") pod \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\" (UID: \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\") " Jul 02 19:36:07.299946 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:07.299803 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a03c85-dca8-4d37-9d87-d6b035ac6dec-maas-api-tls\") pod \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\" (UID: \"42a03c85-dca8-4d37-9d87-d6b035ac6dec\") " Jul 02 19:36:07.300148 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:07.300124 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a03c85-dca8-4d37-9d87-d6b035ac6dec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42a03c85-dca8-4d37-9d87-d6b035ac6dec" (UID: "42a03c85-dca8-4d37-9d87-d6b035ac6dec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:36:07.301951 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:07.301930 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a03c85-dca8-4d37-9d87-d6b035ac6dec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42a03c85-dca8-4d37-9d87-d6b035ac6dec" (UID: "42a03c85-dca8-4d37-9d87-d6b035ac6dec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:36:07.302019 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:07.301970 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a03c85-dca8-4d37-9d87-d6b035ac6dec-kube-api-access-xr9tw" (OuterVolumeSpecName: "kube-api-access-xr9tw") pod "42a03c85-dca8-4d37-9d87-d6b035ac6dec" (UID: "42a03c85-dca8-4d37-9d87-d6b035ac6dec"). InnerVolumeSpecName "kube-api-access-xr9tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:36:07.400547 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:07.400505 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr9tw\" (UniqueName: \"kubernetes.io/projected/42a03c85-dca8-4d37-9d87-d6b035ac6dec-kube-api-access-xr9tw\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:36:07.400547 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:07.400544 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a03c85-dca8-4d37-9d87-d6b035ac6dec-openshift-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:36:07.400547 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:07.400554 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42a03c85-dca8-4d37-9d87-d6b035ac6dec-maas-api-tls\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:36:08.084745 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:08.084713 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" event={"ID":"42a03c85-dca8-4d37-9d87-d6b035ac6dec","Type":"ContainerDied","Data":"eb7d0be1d78aef80e47cdd6f890949f8b95952b96e53b969b8134969a0a111e1"} Jul 02 19:36:08.085183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:08.084766 2565 scope.go:117] "RemoveContainer" containerID="090468b3eb2a9ec6ff2bdeb408678a7170a231db37407d0d232d40ea42938761" Jul 02 19:36:08.085183 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:08.084722 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6" Jul 02 19:36:08.108495 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:08.108472 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6"] Jul 02 19:36:08.117785 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:08.117761 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-a3b3e6-f7b57f644-7f7l6"] Jul 02 19:36:08.763479 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:08.763445 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42a03c85-dca8-4d37-9d87-d6b035ac6dec" path="/var/lib/kubelet/pods/42a03c85-dca8-4d37-9d87-d6b035ac6dec/volumes" Jul 02 19:36:12.096354 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:12.096321 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:12.211771 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:12.211737 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:13.048288 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:13.048253 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:13.148558 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:13.148527 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:13.403172 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:13.403077 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:13.554947 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:13.554908 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:13.893533 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:13.893487 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:14.206991 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:14.206896 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:14.791013 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:14.790978 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:17.583976 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.583940 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:17.645238 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.645204 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g"] Jul 02 19:36:17.645544 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.645531 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25d2b826-0c31-4bd2-a2d4-17a095c2443e" containerName="maas-api" Jul 02 19:36:17.645588 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.645547 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25d2b826-0c31-4bd2-a2d4-17a095c2443e" containerName="maas-api" Jul 02 19:36:17.645588 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.645571 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a03c85-dca8-4d37-9d87-d6b035ac6dec" containerName="maas-api" Jul 02 19:36:17.645588 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.645577 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a03c85-dca8-4d37-9d87-d6b035ac6dec" containerName="maas-api" Jul 02 19:36:17.645922 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.645629 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="25d2b826-0c31-4bd2-a2d4-17a095c2443e" containerName="maas-api" Jul 02 19:36:17.645922 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.645642 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a03c85-dca8-4d37-9d87-d6b035ac6dec" containerName="maas-api" Jul 02 19:36:17.650170 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.650153 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:36:17.652753 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.652731 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-9629e8-serving-cert\"" Jul 02 19:36:17.652833 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.652793 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hxm9f\"" Jul 02 19:36:17.663591 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.663551 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g"] Jul 02 19:36:17.679225 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.679197 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:17.691473 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.691445 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a4f073-39ef-4b41-b9aa-227d2889de74-maas-api-tls\") pod \"maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g\" (UID: \"c4a4f073-39ef-4b41-b9aa-227d2889de74\") " pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:36:17.691574 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.691486 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgmw\" (UniqueName: \"kubernetes.io/projected/c4a4f073-39ef-4b41-b9aa-227d2889de74-kube-api-access-bfgmw\") pod \"maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g\" (UID: \"c4a4f073-39ef-4b41-b9aa-227d2889de74\") " pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:36:17.691574 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.691563 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a4f073-39ef-4b41-b9aa-227d2889de74-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g\" (UID: \"c4a4f073-39ef-4b41-b9aa-227d2889de74\") " pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:36:17.759598 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.759568 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:17.792078 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.792045 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a4f073-39ef-4b41-b9aa-227d2889de74-maas-api-tls\") pod \"maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g\" (UID: \"c4a4f073-39ef-4b41-b9aa-227d2889de74\") " pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:36:17.792203 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.792086 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgmw\" (UniqueName: \"kubernetes.io/projected/c4a4f073-39ef-4b41-b9aa-227d2889de74-kube-api-access-bfgmw\") pod \"maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g\" (UID: \"c4a4f073-39ef-4b41-b9aa-227d2889de74\") " pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:36:17.792203 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.792129 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a4f073-39ef-4b41-b9aa-227d2889de74-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g\" (UID: \"c4a4f073-39ef-4b41-b9aa-227d2889de74\") " pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:36:17.792290 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:36:17.792197 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-9629e8-serving-cert: secret "maas-api-e2e-shared-a-9629e8-serving-cert" not found Jul 02 19:36:17.792290 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:36:17.792287 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4a4f073-39ef-4b41-b9aa-227d2889de74-maas-api-tls podName:c4a4f073-39ef-4b41-b9aa-227d2889de74 nodeName:}" failed. No retries permitted until 2026-07-02 19:36:18.292264966 +0000 UTC m=+2008.089135134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4a4f073-39ef-4b41-b9aa-227d2889de74-maas-api-tls") pod "maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" (UID: "c4a4f073-39ef-4b41-b9aa-227d2889de74") : secret "maas-api-e2e-shared-a-9629e8-serving-cert" not found Jul 02 19:36:17.792656 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.792638 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a4f073-39ef-4b41-b9aa-227d2889de74-openshift-service-ca\") pod \"maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g\" (UID: \"c4a4f073-39ef-4b41-b9aa-227d2889de74\") " pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:36:17.808791 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.808764 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgmw\" (UniqueName: \"kubernetes.io/projected/c4a4f073-39ef-4b41-b9aa-227d2889de74-kube-api-access-bfgmw\") pod \"maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g\" (UID: \"c4a4f073-39ef-4b41-b9aa-227d2889de74\") " pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:36:17.856154 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:17.856058 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:18.267998 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:18.267933 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:18.299858 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:18.299806 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a4f073-39ef-4b41-b9aa-227d2889de74-maas-api-tls\") pod \"maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g\" (UID: \"c4a4f073-39ef-4b41-b9aa-227d2889de74\") " pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:36:18.302270 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:18.302250 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a4f073-39ef-4b41-b9aa-227d2889de74-maas-api-tls\") pod \"maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g\" (UID: \"c4a4f073-39ef-4b41-b9aa-227d2889de74\") " pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:36:18.381584 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:18.381552 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:18.567485 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:18.567398 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:36:18.697770 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:18.697746 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g"] Jul 02 19:36:18.700008 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:36:18.699980 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a4f073_39ef_4b41_b9aa_227d2889de74.slice/crio-4403e9db285facd204e83068d87f65047e60d62f56c376c3b9c91ac72b565d20 WatchSource:0}: Error finding container 4403e9db285facd204e83068d87f65047e60d62f56c376c3b9c91ac72b565d20: Status 404 returned error can't find the container with id 4403e9db285facd204e83068d87f65047e60d62f56c376c3b9c91ac72b565d20 Jul 02 19:36:19.133504 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:19.133465 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" event={"ID":"c4a4f073-39ef-4b41-b9aa-227d2889de74","Type":"ContainerStarted","Data":"4403e9db285facd204e83068d87f65047e60d62f56c376c3b9c91ac72b565d20"} Jul 02 19:36:20.139341 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:20.139307 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" event={"ID":"c4a4f073-39ef-4b41-b9aa-227d2889de74","Type":"ContainerStarted","Data":"908f468a55b96be3183fe8f8bb55f24b55680248c7fd0c91dda430c3aac83920"} Jul 02 19:36:20.139787 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:20.139427 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:36:20.159364 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:20.159317 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" podStartSLOduration=2.669471138 podStartE2EDuration="3.15930316s" totalImagesPullingTime="489.832022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:36:20.156984006 +0000 UTC m=+2009.953854207" watchObservedRunningTime="2026-07-02 19:36:20.15930316 +0000 UTC m=+2009.956173351" Jul 02 19:36:20.489362 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:20.489276 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:20.576723 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:20.576689 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:22.083025 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:22.082992 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:22.442046 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:22.441958 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:22.732997 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:22.732923 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:22.901781 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:22.901745 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:25.653193 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:25.653160 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:25.727570 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:25.727532 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47"] Jul 02 19:36:25.731161 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:25.731141 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:36:25.733942 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:25.733919 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-6500fd-serving-cert\"" Jul 02 19:36:25.745829 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:25.745807 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47"] Jul 02 19:36:25.755079 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:25.755041 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:25.767584 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:25.767562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6837d3-28b2-454a-831e-b8be5e7775f7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47\" (UID: \"4d6837d3-28b2-454a-831e-b8be5e7775f7\") " pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:36:25.767693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:25.767607 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2hnr\" (UniqueName: \"kubernetes.io/projected/4d6837d3-28b2-454a-831e-b8be5e7775f7-kube-api-access-m2hnr\") pod \"maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47\" (UID: \"4d6837d3-28b2-454a-831e-b8be5e7775f7\") " pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:36:25.767693 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:25.767667 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6837d3-28b2-454a-831e-b8be5e7775f7-maas-api-tls\") pod \"maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47\" (UID: \"4d6837d3-28b2-454a-831e-b8be5e7775f7\") " pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:36:25.851087 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:25.851050 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:25.868447 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:25.868417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6837d3-28b2-454a-831e-b8be5e7775f7-maas-api-tls\") pod \"maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47\" (UID: \"4d6837d3-28b2-454a-831e-b8be5e7775f7\") " pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:36:25.868577 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:25.868484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6837d3-28b2-454a-831e-b8be5e7775f7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47\" (UID: \"4d6837d3-28b2-454a-831e-b8be5e7775f7\") " pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:36:25.868577 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:25.868526 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2hnr\" (UniqueName: \"kubernetes.io/projected/4d6837d3-28b2-454a-831e-b8be5e7775f7-kube-api-access-m2hnr\") pod \"maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47\" (UID: \"4d6837d3-28b2-454a-831e-b8be5e7775f7\") " pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:36:25.868654 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:36:25.868573 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-6500fd-serving-cert: secret "maas-api-e2e-shared-b-6500fd-serving-cert" not found Jul 02 19:36:25.868654 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:36:25.868642 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d6837d3-28b2-454a-831e-b8be5e7775f7-maas-api-tls podName:4d6837d3-28b2-454a-831e-b8be5e7775f7 nodeName:}" failed. No retries permitted until 2026-07-02 19:36:26.368621671 +0000 UTC m=+2016.165491859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d6837d3-28b2-454a-831e-b8be5e7775f7-maas-api-tls") pod "maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" (UID: "4d6837d3-28b2-454a-831e-b8be5e7775f7") : secret "maas-api-e2e-shared-b-6500fd-serving-cert" not found Jul 02 19:36:25.869177 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:25.869161 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6837d3-28b2-454a-831e-b8be5e7775f7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47\" (UID: \"4d6837d3-28b2-454a-831e-b8be5e7775f7\") " pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:36:25.880422 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:25.880402 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2hnr\" (UniqueName: \"kubernetes.io/projected/4d6837d3-28b2-454a-831e-b8be5e7775f7-kube-api-access-m2hnr\") pod \"maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47\" (UID: \"4d6837d3-28b2-454a-831e-b8be5e7775f7\") " pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:36:26.149055 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:26.149028 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:36:26.216831 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:26.216795 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:26.373304 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:26.373276 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6837d3-28b2-454a-831e-b8be5e7775f7-maas-api-tls\") pod \"maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47\" (UID: \"4d6837d3-28b2-454a-831e-b8be5e7775f7\") " pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:36:26.375804 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:26.375774 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6837d3-28b2-454a-831e-b8be5e7775f7-maas-api-tls\") pod \"maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47\" (UID: \"4d6837d3-28b2-454a-831e-b8be5e7775f7\") " pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:36:26.456446 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:26.456372 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:26.560354 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:26.560325 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:26.642181 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:26.642148 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:36:26.770534 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:26.770513 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47"] Jul 02 19:36:26.772987 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:36:26.772959 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6837d3_28b2_454a_831e_b8be5e7775f7.slice/crio-029a57416e29aca9f355f61ed5d170a9ab4fe5b49c246db3e80fd9e79e5cf07c WatchSource:0}: Error finding container 029a57416e29aca9f355f61ed5d170a9ab4fe5b49c246db3e80fd9e79e5cf07c: Status 404 returned error can't find the container with id 029a57416e29aca9f355f61ed5d170a9ab4fe5b49c246db3e80fd9e79e5cf07c Jul 02 19:36:27.171595 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:27.171553 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" event={"ID":"4d6837d3-28b2-454a-831e-b8be5e7775f7","Type":"ContainerStarted","Data":"029a57416e29aca9f355f61ed5d170a9ab4fe5b49c246db3e80fd9e79e5cf07c"} Jul 02 19:36:28.177544 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:28.177508 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" event={"ID":"4d6837d3-28b2-454a-831e-b8be5e7775f7","Type":"ContainerStarted","Data":"5d3b38e90107730f56221d4ac5510dd8dec4866afc5bf4ac143a13d19db1164d"} Jul 02 19:36:28.177979 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:28.177574 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:36:28.196362 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:28.196315 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" podStartSLOduration=2.7226376549999998 podStartE2EDuration="3.19630157s" totalImagesPullingTime="473.663915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:36:28.194639331 +0000 UTC m=+2017.991509544" watchObservedRunningTime="2026-07-02 19:36:28.19630157 +0000 UTC m=+2017.993171760" Jul 02 19:36:30.147064 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:30.147033 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:32.815305 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:32.815270 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:34.186964 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:34.186928 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:36:39.163681 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:39.163649 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:39.263764 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:39.263728 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:48.285996 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:48.285959 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:36:48.392602 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:36:48.392571 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:37:20.785605 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.785524 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk"] Jul 02 19:37:20.789273 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.789254 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.793301 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.793268 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6500fd\"/\"kube-root-ca.crt\"" Jul 02 19:37:20.793301 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.793273 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6500fd\"/\"auth-test-model-6500fd-kserve-self-signed-certs\"" Jul 02 19:37:20.793475 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.793326 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6500fd\"/\"openshift-service-ca.crt\"" Jul 02 19:37:20.793539 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.793479 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6500fd\"/\"default-dockercfg-bg6l5\"" Jul 02 19:37:20.798994 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.798974 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk"] Jul 02 19:37:20.821047 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.821027 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:37:20.849281 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.849251 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-tmp-dir\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.849386 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.849283 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-home\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.849386 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.849312 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd0b441-7537-4083-ab24-992f2c4f335f-tls-certs\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.849471 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.849404 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-model-cache\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.849471 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.849445 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-dshm\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.849536 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.849480 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldvmf\" (UniqueName: \"kubernetes.io/projected/2bd0b441-7537-4083-ab24-992f2c4f335f-kube-api-access-ldvmf\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.950176 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.950145 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-tmp-dir\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.950325 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.950182 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-home\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.950325 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.950224 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd0b441-7537-4083-ab24-992f2c4f335f-tls-certs\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.950325 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.950251 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-model-cache\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.950325 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.950290 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-dshm\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.950534 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.950505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldvmf\" (UniqueName: \"kubernetes.io/projected/2bd0b441-7537-4083-ab24-992f2c4f335f-kube-api-access-ldvmf\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.950595 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.950543 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-tmp-dir\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.950651 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.950590 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-model-cache\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.950733 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.950710 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-home\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.952604 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.952583 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-dshm\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.952865 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.952823 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd0b441-7537-4083-ab24-992f2c4f335f-tls-certs\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:20.960835 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:20.960817 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldvmf\" (UniqueName: \"kubernetes.io/projected/2bd0b441-7537-4083-ab24-992f2c4f335f-kube-api-access-ldvmf\") pod \"auth-test-model-6500fd-kserve-5864b48b8c-mqnwk\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:21.099549 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:21.099465 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:21.233633 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:21.233609 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk"] Jul 02 19:37:21.236011 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:37:21.235985 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bd0b441_7537_4083_ab24_992f2c4f335f.slice/crio-45e4dd72a7c4f549000bfa40d1f02b6526a048da3b3a7a1573b053be77ebfff3 WatchSource:0}: Error finding container 45e4dd72a7c4f549000bfa40d1f02b6526a048da3b3a7a1573b053be77ebfff3: Status 404 returned error can't find the container with id 45e4dd72a7c4f549000bfa40d1f02b6526a048da3b3a7a1573b053be77ebfff3 Jul 02 19:37:21.410915 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:21.410803 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" event={"ID":"2bd0b441-7537-4083-ab24-992f2c4f335f","Type":"ContainerStarted","Data":"32e6ff3ac80f4dac0ffa99616556b38be2733e470859a796a67bced89ed431be"} Jul 02 19:37:21.410915 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:21.410864 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" event={"ID":"2bd0b441-7537-4083-ab24-992f2c4f335f","Type":"ContainerStarted","Data":"45e4dd72a7c4f549000bfa40d1f02b6526a048da3b3a7a1573b053be77ebfff3"} Jul 02 19:37:21.428736 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:21.428681 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" podStartSLOduration=1.428660852 podStartE2EDuration="1.428660852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:37:21.427832841 +0000 UTC m=+2071.224703049" watchObservedRunningTime="2026-07-02 19:37:21.428660852 +0000 UTC m=+2071.225531048" Jul 02 19:37:31.100024 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:31.099985 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:31.100533 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:31.100066 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:31.112779 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:31.112754 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:31.471130 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:31.471041 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:37:46.614209 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:46.614181 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:37:51.266274 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:51.266244 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:37:51.269265 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:37:51.269244 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:38:09.970092 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:38:09.970060 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:38:10.068282 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:38:10.068242 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:04.727953 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:04.727866 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:19.567926 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:19.567887 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:20.483663 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:20.483623 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk"] Jul 02 19:39:20.506319 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:20.506285 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:39:35.503967 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.503904 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" podUID="2bd0b441-7537-4083-ab24-992f2c4f335f" containerName="main" containerID="cri-o://32e6ff3ac80f4dac0ffa99616556b38be2733e470859a796a67bced89ed431be" gracePeriod=45 Jul 02 19:39:35.749507 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.749486 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:39:35.790452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.790387 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd0b441-7537-4083-ab24-992f2c4f335f-tls-certs\") pod \"2bd0b441-7537-4083-ab24-992f2c4f335f\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " Jul 02 19:39:35.790452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.790420 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-dshm\") pod \"2bd0b441-7537-4083-ab24-992f2c4f335f\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " Jul 02 19:39:35.790452 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.790441 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-home\") pod \"2bd0b441-7537-4083-ab24-992f2c4f335f\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " Jul 02 19:39:35.790704 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.790473 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-model-cache\") pod \"2bd0b441-7537-4083-ab24-992f2c4f335f\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " Jul 02 19:39:35.790704 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.790491 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-tmp-dir\") pod \"2bd0b441-7537-4083-ab24-992f2c4f335f\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " Jul 02 19:39:35.790704 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.790520 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldvmf\" (UniqueName: \"kubernetes.io/projected/2bd0b441-7537-4083-ab24-992f2c4f335f-kube-api-access-ldvmf\") pod \"2bd0b441-7537-4083-ab24-992f2c4f335f\" (UID: \"2bd0b441-7537-4083-ab24-992f2c4f335f\") " Jul 02 19:39:35.790862 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.790719 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-home" (OuterVolumeSpecName: "home") pod "2bd0b441-7537-4083-ab24-992f2c4f335f" (UID: "2bd0b441-7537-4083-ab24-992f2c4f335f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:39:35.790862 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.790737 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2bd0b441-7537-4083-ab24-992f2c4f335f" (UID: "2bd0b441-7537-4083-ab24-992f2c4f335f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:39:35.790862 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.790814 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2bd0b441-7537-4083-ab24-992f2c4f335f" (UID: "2bd0b441-7537-4083-ab24-992f2c4f335f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:39:35.792755 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.792732 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-dshm" (OuterVolumeSpecName: "dshm") pod "2bd0b441-7537-4083-ab24-992f2c4f335f" (UID: "2bd0b441-7537-4083-ab24-992f2c4f335f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:39:35.792881 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.792806 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd0b441-7537-4083-ab24-992f2c4f335f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2bd0b441-7537-4083-ab24-992f2c4f335f" (UID: "2bd0b441-7537-4083-ab24-992f2c4f335f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:39:35.792881 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.792821 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd0b441-7537-4083-ab24-992f2c4f335f-kube-api-access-ldvmf" (OuterVolumeSpecName: "kube-api-access-ldvmf") pod "2bd0b441-7537-4083-ab24-992f2c4f335f" (UID: "2bd0b441-7537-4083-ab24-992f2c4f335f"). InnerVolumeSpecName "kube-api-access-ldvmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:39:35.891896 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.891864 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-home\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:39:35.891896 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.891889 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-model-cache\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:39:35.891896 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.891899 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-tmp-dir\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:39:35.892103 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.891907 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldvmf\" (UniqueName: \"kubernetes.io/projected/2bd0b441-7537-4083-ab24-992f2c4f335f-kube-api-access-ldvmf\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:39:35.892103 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.891916 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd0b441-7537-4083-ab24-992f2c4f335f-tls-certs\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:39:35.892103 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.891924 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd0b441-7537-4083-ab24-992f2c4f335f-dshm\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:39:35.984392 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.984362 2565 generic.go:356] "Generic (PLEG): container finished" podID="2bd0b441-7537-4083-ab24-992f2c4f335f" containerID="32e6ff3ac80f4dac0ffa99616556b38be2733e470859a796a67bced89ed431be" exitCode=0 Jul 02 19:39:35.984532 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.984414 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" Jul 02 19:39:35.984532 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.984453 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" event={"ID":"2bd0b441-7537-4083-ab24-992f2c4f335f","Type":"ContainerDied","Data":"32e6ff3ac80f4dac0ffa99616556b38be2733e470859a796a67bced89ed431be"} Jul 02 19:39:35.984532 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.984500 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk" event={"ID":"2bd0b441-7537-4083-ab24-992f2c4f335f","Type":"ContainerDied","Data":"45e4dd72a7c4f549000bfa40d1f02b6526a048da3b3a7a1573b053be77ebfff3"} Jul 02 19:39:35.984532 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.984520 2565 scope.go:117] "RemoveContainer" containerID="32e6ff3ac80f4dac0ffa99616556b38be2733e470859a796a67bced89ed431be" Jul 02 19:39:35.993226 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.993207 2565 scope.go:117] "RemoveContainer" containerID="32e6ff3ac80f4dac0ffa99616556b38be2733e470859a796a67bced89ed431be" Jul 02 19:39:35.993496 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:39:35.993477 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e6ff3ac80f4dac0ffa99616556b38be2733e470859a796a67bced89ed431be\": container with ID starting with 32e6ff3ac80f4dac0ffa99616556b38be2733e470859a796a67bced89ed431be not found: ID does not exist" containerID="32e6ff3ac80f4dac0ffa99616556b38be2733e470859a796a67bced89ed431be" Jul 02 19:39:35.993545 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:35.993505 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e6ff3ac80f4dac0ffa99616556b38be2733e470859a796a67bced89ed431be"} err="failed to get container status \"32e6ff3ac80f4dac0ffa99616556b38be2733e470859a796a67bced89ed431be\": rpc error: code = NotFound desc = could not find container \"32e6ff3ac80f4dac0ffa99616556b38be2733e470859a796a67bced89ed431be\": container with ID starting with 32e6ff3ac80f4dac0ffa99616556b38be2733e470859a796a67bced89ed431be not found: ID does not exist" Jul 02 19:39:36.005017 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:36.004995 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk"] Jul 02 19:39:36.008707 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:36.008688 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/auth-test-model-6500fd-kserve-5864b48b8c-mqnwk"] Jul 02 19:39:36.763153 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:36.763123 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bd0b441-7537-4083-ab24-992f2c4f335f" path="/var/lib/kubelet/pods/2bd0b441-7537-4083-ab24-992f2c4f335f/volumes" Jul 02 19:39:37.708278 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:39:37.708240 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:40:02.493487 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.493450 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn"] Jul 02 19:40:02.493956 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.493795 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bd0b441-7537-4083-ab24-992f2c4f335f" containerName="main" Jul 02 19:40:02.493956 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.493805 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bd0b441-7537-4083-ab24-992f2c4f335f" containerName="main" Jul 02 19:40:02.493956 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.493875 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bd0b441-7537-4083-ab24-992f2c4f335f" containerName="main" Jul 02 19:40:02.496666 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.496651 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.500189 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.500152 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6500fd\"/\"sub-test-model-6500fd-kserve-self-signed-certs\"" Jul 02 19:40:02.500189 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.500153 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6500fd\"/\"openshift-service-ca.crt\"" Jul 02 19:40:02.500380 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.500159 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6500fd\"/\"default-dockercfg-bg6l5\"" Jul 02 19:40:02.500380 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.500163 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6500fd\"/\"kube-root-ca.crt\"" Jul 02 19:40:02.504815 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.504793 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn"] Jul 02 19:40:02.518369 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.518345 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:40:02.622900 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.622835 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-dshm\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.623069 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.622956 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-home\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.623069 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.623010 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-tmp-dir\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.623069 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.623032 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515261d5-44cd-4fa1-ba67-bf374f72e22e-tls-certs\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.623187 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.623071 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhm4p\" (UniqueName: \"kubernetes.io/projected/515261d5-44cd-4fa1-ba67-bf374f72e22e-kube-api-access-qhm4p\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.623187 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.623141 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-model-cache\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.724145 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.724103 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-dshm\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.724316 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.724158 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-home\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.724316 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.724179 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-tmp-dir\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.724316 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.724200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515261d5-44cd-4fa1-ba67-bf374f72e22e-tls-certs\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.724316 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.724253 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhm4p\" (UniqueName: \"kubernetes.io/projected/515261d5-44cd-4fa1-ba67-bf374f72e22e-kube-api-access-qhm4p\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.724509 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.724410 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-model-cache\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.724745 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.724647 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-home\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.724745 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.724688 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-tmp-dir\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.724745 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.724716 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-model-cache\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.726522 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.726494 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-dshm\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.726933 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.726914 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515261d5-44cd-4fa1-ba67-bf374f72e22e-tls-certs\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.732229 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.732207 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhm4p\" (UniqueName: \"kubernetes.io/projected/515261d5-44cd-4fa1-ba67-bf374f72e22e-kube-api-access-qhm4p\") pod \"sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.806573 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.806545 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:02.940396 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:02.940369 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn"] Jul 02 19:40:02.942479 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:40:02.942456 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod515261d5_44cd_4fa1_ba67_bf374f72e22e.slice/crio-b1e870df53960cb82b040032cc9c6cc4286d9801c7b411b5358c105ddd9f7083 WatchSource:0}: Error finding container b1e870df53960cb82b040032cc9c6cc4286d9801c7b411b5358c105ddd9f7083: Status 404 returned error can't find the container with id b1e870df53960cb82b040032cc9c6cc4286d9801c7b411b5358c105ddd9f7083 Jul 02 19:40:03.106870 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:03.106751 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" event={"ID":"515261d5-44cd-4fa1-ba67-bf374f72e22e","Type":"ContainerStarted","Data":"d5fabe31305a692205d649a2026bbdbe2381a75c621dd781b71fcb5190c96f2c"} Jul 02 19:40:03.106870 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:03.106794 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" event={"ID":"515261d5-44cd-4fa1-ba67-bf374f72e22e","Type":"ContainerStarted","Data":"b1e870df53960cb82b040032cc9c6cc4286d9801c7b411b5358c105ddd9f7083"} Jul 02 19:40:03.124990 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:03.124936 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" podStartSLOduration=1.124922513 podStartE2EDuration="1.124922513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:40:03.124173116 +0000 UTC m=+2232.921043308" watchObservedRunningTime="2026-07-02 19:40:03.124922513 +0000 UTC m=+2232.921792704" Jul 02 19:40:12.807731 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:12.807637 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:12.807731 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:12.807699 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:12.820446 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:12.820421 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:13.160132 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:13.160048 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:40:45.080323 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:45.080293 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:40:45.889683 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:45.889648 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn"] Jul 02 19:40:46.369861 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:40:46.369810 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:00.910256 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:00.910207 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" podUID="515261d5-44cd-4fa1-ba67-bf374f72e22e" containerName="main" containerID="cri-o://d5fabe31305a692205d649a2026bbdbe2381a75c621dd781b71fcb5190c96f2c" gracePeriod=45 Jul 02 19:41:01.146308 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.146283 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:41:01.227466 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.227387 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-model-cache\") pod \"515261d5-44cd-4fa1-ba67-bf374f72e22e\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " Jul 02 19:41:01.227466 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.227420 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-home\") pod \"515261d5-44cd-4fa1-ba67-bf374f72e22e\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " Jul 02 19:41:01.227466 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.227445 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhm4p\" (UniqueName: \"kubernetes.io/projected/515261d5-44cd-4fa1-ba67-bf374f72e22e-kube-api-access-qhm4p\") pod \"515261d5-44cd-4fa1-ba67-bf374f72e22e\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " Jul 02 19:41:01.227747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.227493 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-tmp-dir\") pod \"515261d5-44cd-4fa1-ba67-bf374f72e22e\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " Jul 02 19:41:01.227747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.227513 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515261d5-44cd-4fa1-ba67-bf374f72e22e-tls-certs\") pod \"515261d5-44cd-4fa1-ba67-bf374f72e22e\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " Jul 02 19:41:01.227747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.227538 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-dshm\") pod \"515261d5-44cd-4fa1-ba67-bf374f72e22e\" (UID: \"515261d5-44cd-4fa1-ba67-bf374f72e22e\") " Jul 02 19:41:01.227747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.227699 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-model-cache" (OuterVolumeSpecName: "model-cache") pod "515261d5-44cd-4fa1-ba67-bf374f72e22e" (UID: "515261d5-44cd-4fa1-ba67-bf374f72e22e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:41:01.227747 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.227709 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-home" (OuterVolumeSpecName: "home") pod "515261d5-44cd-4fa1-ba67-bf374f72e22e" (UID: "515261d5-44cd-4fa1-ba67-bf374f72e22e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:41:01.227955 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.227788 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-model-cache\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:41:01.227955 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.227799 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "515261d5-44cd-4fa1-ba67-bf374f72e22e" (UID: "515261d5-44cd-4fa1-ba67-bf374f72e22e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:41:01.227955 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.227807 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-home\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:41:01.229865 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.229827 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-dshm" (OuterVolumeSpecName: "dshm") pod "515261d5-44cd-4fa1-ba67-bf374f72e22e" (UID: "515261d5-44cd-4fa1-ba67-bf374f72e22e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:41:01.229865 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.229831 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/515261d5-44cd-4fa1-ba67-bf374f72e22e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "515261d5-44cd-4fa1-ba67-bf374f72e22e" (UID: "515261d5-44cd-4fa1-ba67-bf374f72e22e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:41:01.229963 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.229942 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515261d5-44cd-4fa1-ba67-bf374f72e22e-kube-api-access-qhm4p" (OuterVolumeSpecName: "kube-api-access-qhm4p") pod "515261d5-44cd-4fa1-ba67-bf374f72e22e" (UID: "515261d5-44cd-4fa1-ba67-bf374f72e22e"). InnerVolumeSpecName "kube-api-access-qhm4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:41:01.329271 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.329237 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhm4p\" (UniqueName: \"kubernetes.io/projected/515261d5-44cd-4fa1-ba67-bf374f72e22e-kube-api-access-qhm4p\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:41:01.329271 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.329267 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-tmp-dir\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:41:01.329271 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.329277 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/515261d5-44cd-4fa1-ba67-bf374f72e22e-tls-certs\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:41:01.329499 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.329284 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/515261d5-44cd-4fa1-ba67-bf374f72e22e-dshm\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:41:01.354493 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.354458 2565 generic.go:356] "Generic (PLEG): container finished" podID="515261d5-44cd-4fa1-ba67-bf374f72e22e" containerID="d5fabe31305a692205d649a2026bbdbe2381a75c621dd781b71fcb5190c96f2c" exitCode=0 Jul 02 19:41:01.354636 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.354502 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" Jul 02 19:41:01.354636 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.354546 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" event={"ID":"515261d5-44cd-4fa1-ba67-bf374f72e22e","Type":"ContainerDied","Data":"d5fabe31305a692205d649a2026bbdbe2381a75c621dd781b71fcb5190c96f2c"} Jul 02 19:41:01.354636 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.354593 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn" event={"ID":"515261d5-44cd-4fa1-ba67-bf374f72e22e","Type":"ContainerDied","Data":"b1e870df53960cb82b040032cc9c6cc4286d9801c7b411b5358c105ddd9f7083"} Jul 02 19:41:01.354636 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.354618 2565 scope.go:117] "RemoveContainer" containerID="d5fabe31305a692205d649a2026bbdbe2381a75c621dd781b71fcb5190c96f2c" Jul 02 19:41:01.363541 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.363524 2565 scope.go:117] "RemoveContainer" containerID="d5fabe31305a692205d649a2026bbdbe2381a75c621dd781b71fcb5190c96f2c" Jul 02 19:41:01.363779 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:41:01.363761 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5fabe31305a692205d649a2026bbdbe2381a75c621dd781b71fcb5190c96f2c\": container with ID starting with d5fabe31305a692205d649a2026bbdbe2381a75c621dd781b71fcb5190c96f2c not found: ID does not exist" containerID="d5fabe31305a692205d649a2026bbdbe2381a75c621dd781b71fcb5190c96f2c" Jul 02 19:41:01.363830 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.363790 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fabe31305a692205d649a2026bbdbe2381a75c621dd781b71fcb5190c96f2c"} err="failed to get container status \"d5fabe31305a692205d649a2026bbdbe2381a75c621dd781b71fcb5190c96f2c\": rpc error: code = NotFound desc = could not find container \"d5fabe31305a692205d649a2026bbdbe2381a75c621dd781b71fcb5190c96f2c\": container with ID starting with d5fabe31305a692205d649a2026bbdbe2381a75c621dd781b71fcb5190c96f2c not found: ID does not exist" Jul 02 19:41:01.375352 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.375322 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn"] Jul 02 19:41:01.377389 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:01.377367 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/sub-test-model-6500fd-kserve-7c4cb7bf57-f27vn"] Jul 02 19:41:02.763962 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:02.763933 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="515261d5-44cd-4fa1-ba67-bf374f72e22e" path="/var/lib/kubelet/pods/515261d5-44cd-4fa1-ba67-bf374f72e22e/volumes" Jul 02 19:41:12.886469 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:12.886438 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:12.988462 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:12.988429 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:13.345096 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:13.345067 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:26.817095 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:26.817064 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:26.879334 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:26.879307 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:39.973965 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:39.973931 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:40.073186 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:40.073147 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:47.496506 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:47.496414 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:49.149153 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.149095 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:49.245289 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.245256 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:49.497608 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.497531 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m"] Jul 02 19:41:49.497895 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.497881 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515261d5-44cd-4fa1-ba67-bf374f72e22e" containerName="main" Jul 02 19:41:49.497950 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.497896 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="515261d5-44cd-4fa1-ba67-bf374f72e22e" containerName="main" Jul 02 19:41:49.498001 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.497961 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="515261d5-44cd-4fa1-ba67-bf374f72e22e" containerName="main" Jul 02 19:41:49.500465 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.500448 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.503246 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.503227 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6500fd\"/\"kube-root-ca.crt\"" Jul 02 19:41:49.504538 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.504482 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6500fd\"/\"rate-test-model-6500fd-kserve-self-signed-certs\"" Jul 02 19:41:49.504677 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.504558 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-6500fd\"/\"default-dockercfg-bg6l5\"" Jul 02 19:41:49.504677 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.504574 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-6500fd\"/\"openshift-service-ca.crt\"" Jul 02 19:41:49.512709 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.512689 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m"] Jul 02 19:41:49.522314 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.522295 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:41:49.643758 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.643719 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6272121-5ec7-431d-8b10-7d5775bcdaf1-tls-certs\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.643758 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.643757 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-model-cache\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.643998 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.643783 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-dshm\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.643998 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.643878 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bvjz\" (UniqueName: \"kubernetes.io/projected/b6272121-5ec7-431d-8b10-7d5775bcdaf1-kube-api-access-5bvjz\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.643998 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.643929 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-tmp-dir\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.643998 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.643955 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-home\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.745228 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.745185 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bvjz\" (UniqueName: \"kubernetes.io/projected/b6272121-5ec7-431d-8b10-7d5775bcdaf1-kube-api-access-5bvjz\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.745413 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.745238 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-tmp-dir\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.745413 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.745258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-home\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.745413 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.745315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6272121-5ec7-431d-8b10-7d5775bcdaf1-tls-certs\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.745413 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.745335 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-model-cache\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.745413 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.745370 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-dshm\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.745691 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.745664 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-tmp-dir\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.745754 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.745690 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-home\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.745809 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.745768 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-model-cache\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.747944 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.747886 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-dshm\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.748086 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.748070 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6272121-5ec7-431d-8b10-7d5775bcdaf1-tls-certs\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.753425 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.753406 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bvjz\" (UniqueName: \"kubernetes.io/projected/b6272121-5ec7-431d-8b10-7d5775bcdaf1-kube-api-access-5bvjz\") pod \"rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.810221 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.810177 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:49.946083 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:49.946058 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m"] Jul 02 19:41:49.948026 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:41:49.948000 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6272121_5ec7_431d_8b10_7d5775bcdaf1.slice/crio-5286c4ee2abf0f836a2b856b37de878135ab924da3e1333b4e0fe7f70d63aa97 WatchSource:0}: Error finding container 5286c4ee2abf0f836a2b856b37de878135ab924da3e1333b4e0fe7f70d63aa97: Status 404 returned error can't find the container with id 5286c4ee2abf0f836a2b856b37de878135ab924da3e1333b4e0fe7f70d63aa97 Jul 02 19:41:50.566769 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:50.566732 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" event={"ID":"b6272121-5ec7-431d-8b10-7d5775bcdaf1","Type":"ContainerStarted","Data":"3bd33c8792e3d183dd0a94d61886b1d041f03c9919eea7d0db410c833e9314bb"} Jul 02 19:41:50.566769 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:50.566775 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" event={"ID":"b6272121-5ec7-431d-8b10-7d5775bcdaf1","Type":"ContainerStarted","Data":"5286c4ee2abf0f836a2b856b37de878135ab924da3e1333b4e0fe7f70d63aa97"} Jul 02 19:41:50.584389 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:50.584344 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" podStartSLOduration=1.58432992 podStartE2EDuration="1.58432992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:41:50.583151721 +0000 UTC m=+2340.380021912" watchObservedRunningTime="2026-07-02 19:41:50.58432992 +0000 UTC m=+2340.381200111" Jul 02 19:41:59.810338 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:59.810298 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:59.810816 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:59.810362 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:41:59.822741 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:41:59.822716 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:42:00.624930 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:00.624900 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:42:34.642504 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:34.642467 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:35.496350 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:35.496321 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m"] Jul 02 19:42:35.516267 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:35.516224 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:50.516555 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.516513 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" podUID="b6272121-5ec7-431d-8b10-7d5775bcdaf1" containerName="main" containerID="cri-o://3bd33c8792e3d183dd0a94d61886b1d041f03c9919eea7d0db410c833e9314bb" gracePeriod=45 Jul 02 19:42:50.608964 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.608914 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" podUID="b6272121-5ec7-431d-8b10-7d5775bcdaf1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": dial tcp 10.134.0.39:8000: connect: connection refused" Jul 02 19:42:50.754027 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.754004 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:42:50.755031 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.755002 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-tmp-dir\") pod \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " Jul 02 19:42:50.755099 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.755069 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-home\") pod \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " Jul 02 19:42:50.755135 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.755098 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-model-cache\") pod \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " Jul 02 19:42:50.755135 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.755130 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-dshm\") pod \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " Jul 02 19:42:50.755208 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.755187 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bvjz\" (UniqueName: \"kubernetes.io/projected/b6272121-5ec7-431d-8b10-7d5775bcdaf1-kube-api-access-5bvjz\") pod \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " Jul 02 19:42:50.755265 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.755245 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6272121-5ec7-431d-8b10-7d5775bcdaf1-tls-certs\") pod \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\" (UID: \"b6272121-5ec7-431d-8b10-7d5775bcdaf1\") " Jul 02 19:42:50.755319 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.755275 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6272121-5ec7-431d-8b10-7d5775bcdaf1" (UID: "b6272121-5ec7-431d-8b10-7d5775bcdaf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:42:50.755374 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.755322 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-home" (OuterVolumeSpecName: "home") pod "b6272121-5ec7-431d-8b10-7d5775bcdaf1" (UID: "b6272121-5ec7-431d-8b10-7d5775bcdaf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:42:50.755374 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.755335 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6272121-5ec7-431d-8b10-7d5775bcdaf1" (UID: "b6272121-5ec7-431d-8b10-7d5775bcdaf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:42:50.755541 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.755517 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-tmp-dir\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:42:50.755541 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.755542 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-home\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:42:50.755716 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.755552 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-model-cache\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:42:50.757213 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.757185 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-dshm" (OuterVolumeSpecName: "dshm") pod "b6272121-5ec7-431d-8b10-7d5775bcdaf1" (UID: "b6272121-5ec7-431d-8b10-7d5775bcdaf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:42:50.757348 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.757325 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6272121-5ec7-431d-8b10-7d5775bcdaf1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6272121-5ec7-431d-8b10-7d5775bcdaf1" (UID: "b6272121-5ec7-431d-8b10-7d5775bcdaf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:42:50.757420 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.757369 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6272121-5ec7-431d-8b10-7d5775bcdaf1-kube-api-access-5bvjz" (OuterVolumeSpecName: "kube-api-access-5bvjz") pod "b6272121-5ec7-431d-8b10-7d5775bcdaf1" (UID: "b6272121-5ec7-431d-8b10-7d5775bcdaf1"). InnerVolumeSpecName "kube-api-access-5bvjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:42:50.825708 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.825670 2565 generic.go:356] "Generic (PLEG): container finished" podID="b6272121-5ec7-431d-8b10-7d5775bcdaf1" containerID="3bd33c8792e3d183dd0a94d61886b1d041f03c9919eea7d0db410c833e9314bb" exitCode=0 Jul 02 19:42:50.825892 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.825712 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" Jul 02 19:42:50.825892 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.825724 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" event={"ID":"b6272121-5ec7-431d-8b10-7d5775bcdaf1","Type":"ContainerDied","Data":"3bd33c8792e3d183dd0a94d61886b1d041f03c9919eea7d0db410c833e9314bb"} Jul 02 19:42:50.825892 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.825764 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m" event={"ID":"b6272121-5ec7-431d-8b10-7d5775bcdaf1","Type":"ContainerDied","Data":"5286c4ee2abf0f836a2b856b37de878135ab924da3e1333b4e0fe7f70d63aa97"} Jul 02 19:42:50.825892 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.825788 2565 scope.go:117] "RemoveContainer" containerID="3bd33c8792e3d183dd0a94d61886b1d041f03c9919eea7d0db410c833e9314bb" Jul 02 19:42:50.833908 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.833890 2565 scope.go:117] "RemoveContainer" containerID="3bd33c8792e3d183dd0a94d61886b1d041f03c9919eea7d0db410c833e9314bb" Jul 02 19:42:50.834159 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:42:50.834144 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bd33c8792e3d183dd0a94d61886b1d041f03c9919eea7d0db410c833e9314bb\": container with ID starting with 3bd33c8792e3d183dd0a94d61886b1d041f03c9919eea7d0db410c833e9314bb not found: ID does not exist" containerID="3bd33c8792e3d183dd0a94d61886b1d041f03c9919eea7d0db410c833e9314bb" Jul 02 19:42:50.834212 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.834166 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bd33c8792e3d183dd0a94d61886b1d041f03c9919eea7d0db410c833e9314bb"} err="failed to get container status \"3bd33c8792e3d183dd0a94d61886b1d041f03c9919eea7d0db410c833e9314bb\": rpc error: code = NotFound desc = could not find container \"3bd33c8792e3d183dd0a94d61886b1d041f03c9919eea7d0db410c833e9314bb\": container with ID starting with 3bd33c8792e3d183dd0a94d61886b1d041f03c9919eea7d0db410c833e9314bb not found: ID does not exist" Jul 02 19:42:50.847066 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.847039 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m"] Jul 02 19:42:50.850701 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.850680 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-6500fd/rate-test-model-6500fd-kserve-7b86fcf85f-ldh5m"] Jul 02 19:42:50.855919 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.855903 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6272121-5ec7-431d-8b10-7d5775bcdaf1-dshm\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:42:50.855919 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.855920 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bvjz\" (UniqueName: \"kubernetes.io/projected/b6272121-5ec7-431d-8b10-7d5775bcdaf1-kube-api-access-5bvjz\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:42:50.856031 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:50.855929 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6272121-5ec7-431d-8b10-7d5775bcdaf1-tls-certs\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:42:51.293774 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:51.293669 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:42:51.305435 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:51.298939 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wx9nx_a3fd3a8f-67dd-41f4-b313-1c855db78fc3/ovn-acl-logging/0.log" Jul 02 19:42:52.763388 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:52.763355 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6272121-5ec7-431d-8b10-7d5775bcdaf1" path="/var/lib/kubelet/pods/b6272121-5ec7-431d-8b10-7d5775bcdaf1/volumes" Jul 02 19:42:58.321575 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:58.321543 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:58.711131 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:58.711045 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:58.809325 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:58.809267 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:58.960925 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:58.960892 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g"] Jul 02 19:42:58.961130 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:58.961109 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" podUID="c4a4f073-39ef-4b41-b9aa-227d2889de74" containerName="maas-api" containerID="cri-o://908f468a55b96be3183fe8f8bb55f24b55680248c7fd0c91dda430c3aac83920" gracePeriod=30 Jul 02 19:42:59.120896 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.120827 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:59.207648 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.207626 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:42:59.219636 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.219569 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:42:59.225210 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.225187 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a4f073-39ef-4b41-b9aa-227d2889de74-maas-api-tls\") pod \"c4a4f073-39ef-4b41-b9aa-227d2889de74\" (UID: \"c4a4f073-39ef-4b41-b9aa-227d2889de74\") " Jul 02 19:42:59.225352 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.225230 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfgmw\" (UniqueName: \"kubernetes.io/projected/c4a4f073-39ef-4b41-b9aa-227d2889de74-kube-api-access-bfgmw\") pod \"c4a4f073-39ef-4b41-b9aa-227d2889de74\" (UID: \"c4a4f073-39ef-4b41-b9aa-227d2889de74\") " Jul 02 19:42:59.225352 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.225259 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a4f073-39ef-4b41-b9aa-227d2889de74-openshift-service-ca\") pod \"c4a4f073-39ef-4b41-b9aa-227d2889de74\" (UID: \"c4a4f073-39ef-4b41-b9aa-227d2889de74\") " Jul 02 19:42:59.225941 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.225893 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a4f073-39ef-4b41-b9aa-227d2889de74-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c4a4f073-39ef-4b41-b9aa-227d2889de74" (UID: "c4a4f073-39ef-4b41-b9aa-227d2889de74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:42:59.228078 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.228056 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4a4f073-39ef-4b41-b9aa-227d2889de74-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c4a4f073-39ef-4b41-b9aa-227d2889de74" (UID: "c4a4f073-39ef-4b41-b9aa-227d2889de74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:42:59.228171 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.228084 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a4f073-39ef-4b41-b9aa-227d2889de74-kube-api-access-bfgmw" (OuterVolumeSpecName: "kube-api-access-bfgmw") pod "c4a4f073-39ef-4b41-b9aa-227d2889de74" (UID: "c4a4f073-39ef-4b41-b9aa-227d2889de74"). InnerVolumeSpecName "kube-api-access-bfgmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:42:59.326434 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.326402 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4a4f073-39ef-4b41-b9aa-227d2889de74-maas-api-tls\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:42:59.326434 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.326430 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfgmw\" (UniqueName: \"kubernetes.io/projected/c4a4f073-39ef-4b41-b9aa-227d2889de74-kube-api-access-bfgmw\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:42:59.326434 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.326439 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a4f073-39ef-4b41-b9aa-227d2889de74-openshift-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:42:59.866609 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.866572 2565 generic.go:356] "Generic (PLEG): container finished" podID="c4a4f073-39ef-4b41-b9aa-227d2889de74" containerID="908f468a55b96be3183fe8f8bb55f24b55680248c7fd0c91dda430c3aac83920" exitCode=0 Jul 02 19:42:59.866609 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.866613 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" Jul 02 19:42:59.866836 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.866666 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" event={"ID":"c4a4f073-39ef-4b41-b9aa-227d2889de74","Type":"ContainerDied","Data":"908f468a55b96be3183fe8f8bb55f24b55680248c7fd0c91dda430c3aac83920"} Jul 02 19:42:59.866836 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.866718 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g" event={"ID":"c4a4f073-39ef-4b41-b9aa-227d2889de74","Type":"ContainerDied","Data":"4403e9db285facd204e83068d87f65047e60d62f56c376c3b9c91ac72b565d20"} Jul 02 19:42:59.866836 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.866747 2565 scope.go:117] "RemoveContainer" containerID="908f468a55b96be3183fe8f8bb55f24b55680248c7fd0c91dda430c3aac83920" Jul 02 19:42:59.876091 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.876073 2565 scope.go:117] "RemoveContainer" containerID="908f468a55b96be3183fe8f8bb55f24b55680248c7fd0c91dda430c3aac83920" Jul 02 19:42:59.876344 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:42:59.876328 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"908f468a55b96be3183fe8f8bb55f24b55680248c7fd0c91dda430c3aac83920\": container with ID starting with 908f468a55b96be3183fe8f8bb55f24b55680248c7fd0c91dda430c3aac83920 not found: ID does not exist" containerID="908f468a55b96be3183fe8f8bb55f24b55680248c7fd0c91dda430c3aac83920" Jul 02 19:42:59.876391 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.876354 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"908f468a55b96be3183fe8f8bb55f24b55680248c7fd0c91dda430c3aac83920"} err="failed to get container status \"908f468a55b96be3183fe8f8bb55f24b55680248c7fd0c91dda430c3aac83920\": rpc error: code = NotFound desc = could not find container \"908f468a55b96be3183fe8f8bb55f24b55680248c7fd0c91dda430c3aac83920\": container with ID starting with 908f468a55b96be3183fe8f8bb55f24b55680248c7fd0c91dda430c3aac83920 not found: ID does not exist" Jul 02 19:42:59.887307 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.887283 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g"] Jul 02 19:42:59.892320 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:42:59.892300 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-9629e8-7fbdbbd9fc-j7n8g"] Jul 02 19:43:00.768463 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:00.768418 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4a4f073-39ef-4b41-b9aa-227d2889de74" path="/var/lib/kubelet/pods/c4a4f073-39ef-4b41-b9aa-227d2889de74/volumes" Jul 02 19:43:04.907007 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:04.906959 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:04.974526 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:04.974479 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:05.183412 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:05.183324 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:05.416061 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:05.416032 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:05.587761 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:05.587731 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:05.997719 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:05.997642 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:06.099234 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.099200 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:06.247457 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.247424 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47"] Jul 02 19:43:06.247698 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.247660 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" podUID="4d6837d3-28b2-454a-831e-b8be5e7775f7" containerName="maas-api" containerID="cri-o://5d3b38e90107730f56221d4ac5510dd8dec4866afc5bf4ac143a13d19db1164d" gracePeriod=30 Jul 02 19:43:06.273257 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.273219 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:06.687332 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.687312 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:43:06.785267 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.785187 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6837d3-28b2-454a-831e-b8be5e7775f7-openshift-service-ca\") pod \"4d6837d3-28b2-454a-831e-b8be5e7775f7\" (UID: \"4d6837d3-28b2-454a-831e-b8be5e7775f7\") " Jul 02 19:43:06.785267 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.785265 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2hnr\" (UniqueName: \"kubernetes.io/projected/4d6837d3-28b2-454a-831e-b8be5e7775f7-kube-api-access-m2hnr\") pod \"4d6837d3-28b2-454a-831e-b8be5e7775f7\" (UID: \"4d6837d3-28b2-454a-831e-b8be5e7775f7\") " Jul 02 19:43:06.785481 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.785290 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6837d3-28b2-454a-831e-b8be5e7775f7-maas-api-tls\") pod \"4d6837d3-28b2-454a-831e-b8be5e7775f7\" (UID: \"4d6837d3-28b2-454a-831e-b8be5e7775f7\") " Jul 02 19:43:06.785542 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.785492 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6837d3-28b2-454a-831e-b8be5e7775f7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d6837d3-28b2-454a-831e-b8be5e7775f7" (UID: "4d6837d3-28b2-454a-831e-b8be5e7775f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:43:06.787474 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.787449 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6837d3-28b2-454a-831e-b8be5e7775f7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d6837d3-28b2-454a-831e-b8be5e7775f7" (UID: "4d6837d3-28b2-454a-831e-b8be5e7775f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:43:06.787572 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.787478 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6837d3-28b2-454a-831e-b8be5e7775f7-kube-api-access-m2hnr" (OuterVolumeSpecName: "kube-api-access-m2hnr") pod "4d6837d3-28b2-454a-831e-b8be5e7775f7" (UID: "4d6837d3-28b2-454a-831e-b8be5e7775f7"). InnerVolumeSpecName "kube-api-access-m2hnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:43:06.886720 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.886681 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2hnr\" (UniqueName: \"kubernetes.io/projected/4d6837d3-28b2-454a-831e-b8be5e7775f7-kube-api-access-m2hnr\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:43:06.886720 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.886717 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6837d3-28b2-454a-831e-b8be5e7775f7-maas-api-tls\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:43:06.886901 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.886731 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6837d3-28b2-454a-831e-b8be5e7775f7-openshift-service-ca\") on node \"ip-10-0-142-150.ec2.internal\" DevicePath \"\"" Jul 02 19:43:06.897945 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.897916 2565 generic.go:356] "Generic (PLEG): container finished" podID="4d6837d3-28b2-454a-831e-b8be5e7775f7" containerID="5d3b38e90107730f56221d4ac5510dd8dec4866afc5bf4ac143a13d19db1164d" exitCode=0 Jul 02 19:43:06.898067 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.897958 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" Jul 02 19:43:06.898067 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.898003 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" event={"ID":"4d6837d3-28b2-454a-831e-b8be5e7775f7","Type":"ContainerDied","Data":"5d3b38e90107730f56221d4ac5510dd8dec4866afc5bf4ac143a13d19db1164d"} Jul 02 19:43:06.898067 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.898050 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47" event={"ID":"4d6837d3-28b2-454a-831e-b8be5e7775f7","Type":"ContainerDied","Data":"029a57416e29aca9f355f61ed5d170a9ab4fe5b49c246db3e80fd9e79e5cf07c"} Jul 02 19:43:06.898215 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.898070 2565 scope.go:117] "RemoveContainer" containerID="5d3b38e90107730f56221d4ac5510dd8dec4866afc5bf4ac143a13d19db1164d" Jul 02 19:43:06.912698 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.912678 2565 scope.go:117] "RemoveContainer" containerID="5d3b38e90107730f56221d4ac5510dd8dec4866afc5bf4ac143a13d19db1164d" Jul 02 19:43:06.912983 ip-10-0-142-150 kubenswrapper[2565]: E0702 19:43:06.912959 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3b38e90107730f56221d4ac5510dd8dec4866afc5bf4ac143a13d19db1164d\": container with ID starting with 5d3b38e90107730f56221d4ac5510dd8dec4866afc5bf4ac143a13d19db1164d not found: ID does not exist" containerID="5d3b38e90107730f56221d4ac5510dd8dec4866afc5bf4ac143a13d19db1164d" Jul 02 19:43:06.913044 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.912994 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3b38e90107730f56221d4ac5510dd8dec4866afc5bf4ac143a13d19db1164d"} err="failed to get container status \"5d3b38e90107730f56221d4ac5510dd8dec4866afc5bf4ac143a13d19db1164d\": rpc error: code = NotFound desc = could not find container \"5d3b38e90107730f56221d4ac5510dd8dec4866afc5bf4ac143a13d19db1164d\": container with ID starting with 5d3b38e90107730f56221d4ac5510dd8dec4866afc5bf4ac143a13d19db1164d not found: ID does not exist" Jul 02 19:43:06.923481 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.923430 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47"] Jul 02 19:43:06.925565 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:06.925545 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-6500fd-6bc86d9948-t6x47"] Jul 02 19:43:08.762943 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:08.762910 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d6837d3-28b2-454a-831e-b8be5e7775f7" path="/var/lib/kubelet/pods/4d6837d3-28b2-454a-831e-b8be5e7775f7/volumes" Jul 02 19:43:12.064879 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:12.064782 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:12.129493 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:12.129464 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:12.351364 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:12.351282 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:12.569981 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:12.569949 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:18.648507 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:18.648479 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cd5847cc4-t4khf_cf3f821b-654f-497b-ae98-9760fb90c79b/copy-manifests/0.log" Jul 02 19:43:18.655047 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:18.655020 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cd5847cc4-t4khf_cf3f821b-654f-497b-ae98-9760fb90c79b/manager/0.log" Jul 02 19:43:21.476884 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:21.476834 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zdhtn_6b4f8cf4-179c-43a6-9631-734bba0356cc/discovery/0.log" Jul 02 19:43:21.591251 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:21.591224 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7656b89f9d-2r6jd_fd02a48a-bea1-4574-a1f7-3427ef175a1a/kube-auth-proxy/0.log" Jul 02 19:43:22.874226 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:22.874200 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbth6cv_90442dac-8e2f-4d15-a56c-2d0c6ef52cb8/main/0.log" Jul 02 19:43:27.504296 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:27.504269 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:27.602967 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:27.602934 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" limit=1024 Jul 02 19:43:30.294375 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:30.294342 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t7n9j_6f01d507-0625-477b-bf2f-7382a4c5bc9b/global-pull-secret-syncer/0.log" Jul 02 19:43:30.364629 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:30.364603 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c485v_05ad455a-193e-433c-994e-44600efc8041/konnectivity-agent/0.log" Jul 02 19:43:30.509877 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:30.509814 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-150.ec2.internal_4dee40681f34b7bc84fa53a0b9970f9c/haproxy/0.log" Jul 02 19:43:38.268204 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.268177 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lr94p_b182dd57-e318-4ba4-8ceb-8e9fb7d85406/node-exporter/0.log" Jul 02 19:43:38.292870 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.292819 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lr94p_b182dd57-e318-4ba4-8ceb-8e9fb7d85406/kube-rbac-proxy/0.log" Jul 02 19:43:38.314549 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.314529 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lr94p_b182dd57-e318-4ba4-8ceb-8e9fb7d85406/init-textfile/0.log" Jul 02 19:43:38.338303 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.338281 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-rr86n_6d036447-e4c8-4d76-a475-d159cba51f95/kube-rbac-proxy-main/0.log" Jul 02 19:43:38.365178 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.365163 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-rr86n_6d036447-e4c8-4d76-a475-d159cba51f95/kube-rbac-proxy-self/0.log" Jul 02 19:43:38.386034 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.385981 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-rr86n_6d036447-e4c8-4d76-a475-d159cba51f95/openshift-state-metrics/0.log" Jul 02 19:43:38.424214 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.424198 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4e118da-3b87-429d-9353-37ba79dc3205/prometheus/0.log" Jul 02 19:43:38.444420 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.444404 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4e118da-3b87-429d-9353-37ba79dc3205/config-reloader/0.log" Jul 02 19:43:38.468589 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.468574 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4e118da-3b87-429d-9353-37ba79dc3205/thanos-sidecar/0.log" Jul 02 19:43:38.490224 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.490206 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4e118da-3b87-429d-9353-37ba79dc3205/kube-rbac-proxy-web/0.log" Jul 02 19:43:38.511713 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.511696 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4e118da-3b87-429d-9353-37ba79dc3205/kube-rbac-proxy/0.log" Jul 02 19:43:38.532463 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.532447 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4e118da-3b87-429d-9353-37ba79dc3205/kube-rbac-proxy-thanos/0.log" Jul 02 19:43:38.555059 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.555041 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d4e118da-3b87-429d-9353-37ba79dc3205/init-config-reloader/0.log" Jul 02 19:43:38.579126 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.579112 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-p2kw7_1072a934-656e-4676-98eb-1d4c432e733c/prometheus-operator/0.log" Jul 02 19:43:38.596829 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.596804 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-p2kw7_1072a934-656e-4676-98eb-1d4c432e733c/kube-rbac-proxy/0.log" Jul 02 19:43:38.737787 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.737709 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c68d759d-mmtpp_e409ca8c-556c-445a-bd68-822247bbd65f/thanos-query/0.log" Jul 02 19:43:38.757550 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.757526 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c68d759d-mmtpp_e409ca8c-556c-445a-bd68-822247bbd65f/kube-rbac-proxy-web/0.log" Jul 02 19:43:38.777691 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.777667 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c68d759d-mmtpp_e409ca8c-556c-445a-bd68-822247bbd65f/kube-rbac-proxy/0.log" Jul 02 19:43:38.795785 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.795767 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c68d759d-mmtpp_e409ca8c-556c-445a-bd68-822247bbd65f/prom-label-proxy/0.log" Jul 02 19:43:38.813922 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.813905 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c68d759d-mmtpp_e409ca8c-556c-445a-bd68-822247bbd65f/kube-rbac-proxy-rules/0.log" Jul 02 19:43:38.832815 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:38.832796 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69c68d759d-mmtpp_e409ca8c-556c-445a-bd68-822247bbd65f/kube-rbac-proxy-metrics/0.log" Jul 02 19:43:40.481753 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.481714 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9"] Jul 02 19:43:40.482146 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.482023 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4a4f073-39ef-4b41-b9aa-227d2889de74" containerName="maas-api" Jul 02 19:43:40.482146 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.482034 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4a4f073-39ef-4b41-b9aa-227d2889de74" containerName="maas-api" Jul 02 19:43:40.482146 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.482054 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d6837d3-28b2-454a-831e-b8be5e7775f7" containerName="maas-api" Jul 02 19:43:40.482146 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.482059 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d6837d3-28b2-454a-831e-b8be5e7775f7" containerName="maas-api" Jul 02 19:43:40.482146 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.482071 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6272121-5ec7-431d-8b10-7d5775bcdaf1" containerName="main" Jul 02 19:43:40.482146 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.482076 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6272121-5ec7-431d-8b10-7d5775bcdaf1" containerName="main" Jul 02 19:43:40.482146 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.482119 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d6837d3-28b2-454a-831e-b8be5e7775f7" containerName="maas-api" Jul 02 19:43:40.482146 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.482129 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4a4f073-39ef-4b41-b9aa-227d2889de74" containerName="maas-api" Jul 02 19:43:40.482146 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.482136 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6272121-5ec7-431d-8b10-7d5775bcdaf1" containerName="main" Jul 02 19:43:40.485555 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.485536 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.487912 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.487892 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d52kh\"/\"kube-root-ca.crt\"" Jul 02 19:43:40.488011 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.487934 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d52kh\"/\"openshift-service-ca.crt\"" Jul 02 19:43:40.488982 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.488968 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d52kh\"/\"default-dockercfg-n4n8v\"" Jul 02 19:43:40.491614 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.491593 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9"] Jul 02 19:43:40.679240 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.679188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ctbj\" (UniqueName: \"kubernetes.io/projected/41cedee1-8e75-4871-9e33-5939c2929968-kube-api-access-9ctbj\") pod \"perf-node-gather-daemonset-dlvl9\" (UID: \"41cedee1-8e75-4871-9e33-5939c2929968\") " pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.679417 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.679268 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41cedee1-8e75-4871-9e33-5939c2929968-podres\") pod \"perf-node-gather-daemonset-dlvl9\" (UID: \"41cedee1-8e75-4871-9e33-5939c2929968\") " pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.679417 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.679326 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41cedee1-8e75-4871-9e33-5939c2929968-sys\") pod \"perf-node-gather-daemonset-dlvl9\" (UID: \"41cedee1-8e75-4871-9e33-5939c2929968\") " pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.679417 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.679361 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41cedee1-8e75-4871-9e33-5939c2929968-lib-modules\") pod \"perf-node-gather-daemonset-dlvl9\" (UID: \"41cedee1-8e75-4871-9e33-5939c2929968\") " pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.679417 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.679400 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41cedee1-8e75-4871-9e33-5939c2929968-proc\") pod \"perf-node-gather-daemonset-dlvl9\" (UID: \"41cedee1-8e75-4871-9e33-5939c2929968\") " pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.779802 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.779774 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41cedee1-8e75-4871-9e33-5939c2929968-proc\") pod \"perf-node-gather-daemonset-dlvl9\" (UID: \"41cedee1-8e75-4871-9e33-5939c2929968\") " pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.779982 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.779806 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ctbj\" (UniqueName: \"kubernetes.io/projected/41cedee1-8e75-4871-9e33-5939c2929968-kube-api-access-9ctbj\") pod \"perf-node-gather-daemonset-dlvl9\" (UID: \"41cedee1-8e75-4871-9e33-5939c2929968\") " pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.779982 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.779837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41cedee1-8e75-4871-9e33-5939c2929968-podres\") pod \"perf-node-gather-daemonset-dlvl9\" (UID: \"41cedee1-8e75-4871-9e33-5939c2929968\") " pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.779982 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.779910 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/41cedee1-8e75-4871-9e33-5939c2929968-proc\") pod \"perf-node-gather-daemonset-dlvl9\" (UID: \"41cedee1-8e75-4871-9e33-5939c2929968\") " pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.779982 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.779917 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41cedee1-8e75-4871-9e33-5939c2929968-sys\") pod \"perf-node-gather-daemonset-dlvl9\" (UID: \"41cedee1-8e75-4871-9e33-5939c2929968\") " pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.779982 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.779965 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41cedee1-8e75-4871-9e33-5939c2929968-lib-modules\") pod \"perf-node-gather-daemonset-dlvl9\" (UID: \"41cedee1-8e75-4871-9e33-5939c2929968\") " pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.780178 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.779993 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/41cedee1-8e75-4871-9e33-5939c2929968-sys\") pod \"perf-node-gather-daemonset-dlvl9\" (UID: \"41cedee1-8e75-4871-9e33-5939c2929968\") " pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.780178 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.780009 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/41cedee1-8e75-4871-9e33-5939c2929968-podres\") pod \"perf-node-gather-daemonset-dlvl9\" (UID: \"41cedee1-8e75-4871-9e33-5939c2929968\") " pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.780178 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.780097 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/41cedee1-8e75-4871-9e33-5939c2929968-lib-modules\") pod \"perf-node-gather-daemonset-dlvl9\" (UID: \"41cedee1-8e75-4871-9e33-5939c2929968\") " pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.787714 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.787692 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ctbj\" (UniqueName: \"kubernetes.io/projected/41cedee1-8e75-4871-9e33-5939c2929968-kube-api-access-9ctbj\") pod \"perf-node-gather-daemonset-dlvl9\" (UID: \"41cedee1-8e75-4871-9e33-5939c2929968\") " pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.796538 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.796521 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:40.922702 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:40.922673 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9"] Jul 02 19:43:40.925191 ip-10-0-142-150 kubenswrapper[2565]: W0702 19:43:40.925166 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod41cedee1_8e75_4871_9e33_5939c2929968.slice/crio-0b572e6c927633efd5ee7bc4802facfb2924c01514aadd1a4925a2d273128382 WatchSource:0}: Error finding container 0b572e6c927633efd5ee7bc4802facfb2924c01514aadd1a4925a2d273128382: Status 404 returned error can't find the container with id 0b572e6c927633efd5ee7bc4802facfb2924c01514aadd1a4925a2d273128382 Jul 02 19:43:41.045589 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:41.045523 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" event={"ID":"41cedee1-8e75-4871-9e33-5939c2929968","Type":"ContainerStarted","Data":"83d3d490173d3bce409574b5d1f573d787bf29daa9144f7fffc53019d0d10776"} Jul 02 19:43:41.045589 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:41.045571 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" event={"ID":"41cedee1-8e75-4871-9e33-5939c2929968","Type":"ContainerStarted","Data":"0b572e6c927633efd5ee7bc4802facfb2924c01514aadd1a4925a2d273128382"} Jul 02 19:43:41.045717 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:41.045689 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:41.066394 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:41.066353 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" podStartSLOduration=1.066340206 podStartE2EDuration="1.066340206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:43:41.066075323 +0000 UTC m=+2450.862945515" watchObservedRunningTime="2026-07-02 19:43:41.066340206 +0000 UTC m=+2450.863210396" Jul 02 19:43:41.208070 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:41.208036 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8ffd9df7b-j28nh_65e458c3-9c44-4f8b-8dcb-911ffbcadf69/console/0.log" Jul 02 19:43:43.088165 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:43.088138 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d65jh_6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56/dns/0.log" Jul 02 19:43:43.115742 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:43.115720 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d65jh_6ce2b18f-f9a2-46aa-bd2a-6f6aaccddb56/kube-rbac-proxy/0.log" Jul 02 19:43:43.246539 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:43.246514 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vd52d_57a35ed1-ba9b-41d9-bf9b-bd99874366c6/dns-node-resolver/0.log" Jul 02 19:43:43.823259 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:43.823230 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rkjd9_35b8716c-c781-4444-85ea-b906b33f52fb/node-ca/0.log" Jul 02 19:43:44.974561 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:44.974529 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zdhtn_6b4f8cf4-179c-43a6-9631-734bba0356cc/discovery/0.log" Jul 02 19:43:44.993911 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:44.993884 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7656b89f9d-2r6jd_fd02a48a-bea1-4574-a1f7-3427ef175a1a/kube-auth-proxy/0.log" Jul 02 19:43:45.934178 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:45.934145 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rscwv_28fbcbc4-81c8-4e67-b448-e912dd00eb27/serve-healthcheck-canary/0.log" Jul 02 19:43:46.492994 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:46.492964 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hfq7v_0f63be4f-6f08-489d-bc63-b33240cf694b/kube-rbac-proxy/0.log" Jul 02 19:43:46.513378 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:46.513354 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hfq7v_0f63be4f-6f08-489d-bc63-b33240cf694b/exporter/0.log" Jul 02 19:43:46.533245 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:46.533208 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hfq7v_0f63be4f-6f08-489d-bc63-b33240cf694b/extractor/0.log" Jul 02 19:43:47.060509 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:47.060477 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-d52kh/perf-node-gather-daemonset-dlvl9" Jul 02 19:43:49.485974 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:49.485946 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cd5847cc4-t4khf_cf3f821b-654f-497b-ae98-9760fb90c79b/manager/0.log" Jul 02 19:43:49.507791 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:49.507763 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cd5847cc4-t4khf_cf3f821b-654f-497b-ae98-9760fb90c79b/copy-manifests/0.log" Jul 02 19:43:51.109142 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:51.109112 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6d578bc98c-8q47r_73a1c135-f1cc-4042-87ca-010ee4ab45b6/manager/0.log" Jul 02 19:43:51.157550 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:51.157516 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-qxxzl_5f912e6c-1f59-482a-a0cb-d91a4ff1a2c6/openshift-lws-operator/0.log" Jul 02 19:43:51.534702 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:51.534665 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zdhtn_6b4f8cf4-179c-43a6-9631-734bba0356cc/discovery/0.log" Jul 02 19:43:51.546326 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:51.546308 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7656b89f9d-2r6jd_fd02a48a-bea1-4574-a1f7-3427ef175a1a/kube-auth-proxy/0.log" Jul 02 19:43:52.946439 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:52.946404 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zdhtn_6b4f8cf4-179c-43a6-9631-734bba0356cc/discovery/0.log" Jul 02 19:43:52.957912 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:52.957875 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7656b89f9d-2r6jd_fd02a48a-bea1-4574-a1f7-3427ef175a1a/kube-auth-proxy/0.log" Jul 02 19:43:57.818519 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:57.818488 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kdm4z_7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab/kube-multus-additional-cni-plugins/0.log" Jul 02 19:43:57.839799 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:57.839773 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kdm4z_7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab/egress-router-binary-copy/0.log" Jul 02 19:43:57.859331 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:57.859310 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kdm4z_7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab/cni-plugins/0.log" Jul 02 19:43:57.879058 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:57.879041 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kdm4z_7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab/bond-cni-plugin/0.log" Jul 02 19:43:57.901559 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:57.901534 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kdm4z_7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab/routeoverride-cni/0.log" Jul 02 19:43:57.922440 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:57.922422 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kdm4z_7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab/whereabouts-cni-bincopy/0.log" Jul 02 19:43:57.941813 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:57.941793 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kdm4z_7c19a6c2-f663-4fd4-a5f9-5c39a74c26ab/whereabouts-cni/0.log" Jul 02 19:43:58.005269 ip-10-0-142-150 kubenswrapper[2565]: I0702 19:43:58.005248 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zxctb_f583ae21-02a0-4d33-b57c-b8370bbaecdc/kube-multus/0.log"