Jul 23 00:21:24.521281 ip-10-0-139-1 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 23 00:21:24.521294 ip-10-0-139-1 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 23 00:21:24.521322 ip-10-0-139-1 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 23 00:21:24.521582 ip-10-0-139-1 systemd[1]: Failed to start Kubernetes Kubelet. Jul 23 00:21:34.689558 ip-10-0-139-1 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 23 00:21:34.689576 ip-10-0-139-1 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot de0c1cd67e6d46e39d98ffd20f54f77f -- Jul 23 00:23:27.388805 ip-10-0-139-1 systemd[1]: Starting Kubernetes Kubelet... Jul 23 00:23:27.864462 ip-10-0-139-1 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 00:23:27.864462 ip-10-0-139-1 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 00:23:27.864462 ip-10-0-139-1 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 00:23:27.864462 ip-10-0-139-1 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 00:23:27.864462 ip-10-0-139-1 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 00:23:27.864462 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.798646 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 00:23:27.864462 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806124 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:23:27.864462 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806136 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:23:27.864462 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806141 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:23:27.864462 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806144 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:23:27.864462 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806147 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:23:27.864462 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806150 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:23:27.864462 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806153 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:23:27.864462 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806156 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806158 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806161 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806166 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806170 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806173 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806176 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806179 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806181 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806185 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806189 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806193 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806202 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806206 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806209 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806212 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806214 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806217 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806220 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:23:27.865280 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806223 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806226 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806230 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806233 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806236 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806238 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806241 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806243 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806246 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806250 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806253 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806255 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806259 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806261 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806264 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806266 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806270 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806273 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806276 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806278 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:23:27.866098 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806281 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806284 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806287 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806289 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806292 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806295 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806297 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806300 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806302 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806305 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806307 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806310 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806312 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806315 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806318 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806322 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806325 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806328 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806330 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806333 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:23:27.866956 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806335 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806338 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806341 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806344 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806347 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806350 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806352 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806355 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806358 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806361 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806363 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806366 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806369 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806372 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806374 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806377 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806379 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806382 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806384 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806387 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:23:27.868019 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806389 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806393 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806396 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806398 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806401 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806404 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806406 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806409 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806411 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806413 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806416 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806418 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.806421 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807122 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807129 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807132 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807135 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807137 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807140 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807143 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:23:27.869294 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807146 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807148 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807151 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807153 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807157 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807160 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807163 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807166 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807168 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807171 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807173 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807176 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807178 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807181 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807184 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807187 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807189 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807192 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807195 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807197 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:23:27.870261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807200 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:23:27.870263 ip-10-0-139-1 systemd[1]: Started Kubernetes Kubelet. Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807202 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807205 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807208 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807211 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807214 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807216 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807219 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807222 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807224 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807227 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807230 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807233 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807235 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807238 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807241 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807246 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807250 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807253 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:23:27.871273 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807256 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807259 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807262 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807265 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807268 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807270 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807273 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807275 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807278 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807280 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807283 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807286 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807289 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807294 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807297 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807300 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807303 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807306 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807309 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:23:27.872486 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807312 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807316 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807320 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807323 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807326 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807329 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807333 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807335 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807338 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807341 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807344 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807347 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807350 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807352 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807355 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807358 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807360 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807363 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807365 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807368 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:23:27.873261 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807370 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807373 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807375 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807378 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807380 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807383 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807386 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807388 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807394 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807396 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807399 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807401 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807404 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807406 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807799 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807804 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807809 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807812 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807816 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807819 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:23:27.873982 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807821 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807824 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807827 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807829 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807832 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807834 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807837 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807839 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807842 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807845 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807848 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807850 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807854 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807856 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807859 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807862 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807865 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807867 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807870 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807872 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:23:27.874687 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807877 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807879 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807882 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807884 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807887 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807889 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807892 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807895 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807898 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807903 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807907 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807910 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807913 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807915 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807918 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807920 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807923 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807925 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807928 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807930 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:23:27.875306 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807933 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807937 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807940 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807943 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807946 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807949 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807952 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807955 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807957 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807960 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807963 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807965 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807971 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807974 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807977 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807980 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807983 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807985 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807988 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:23:27.876014 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807990 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807993 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807996 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.807999 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808002 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808004 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808007 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808009 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808012 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808014 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808017 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808019 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808022 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808024 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808027 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808030 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808032 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808035 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808037 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:23:27.876688 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808040 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808042 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808045 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808047 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808050 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808052 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808057 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808060 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808063 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808065 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808068 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808070 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808073 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808076 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.808078 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808141 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808148 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808154 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808158 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808163 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808166 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 00:23:27.877181 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808170 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808174 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808177 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808180 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808183 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808187 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808190 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808193 2564 flags.go:64] FLAG: --cgroup-root="" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808196 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808198 2564 flags.go:64] FLAG: --client-ca-file="" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808201 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808204 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808209 2564 flags.go:64] FLAG: --cluster-domain="" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808212 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808215 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808218 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808221 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808227 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808230 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808233 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808236 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808239 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808242 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808245 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 00:23:27.877779 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808248 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808251 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808255 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808259 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808262 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808265 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808268 2564 flags.go:64] FLAG: --enable-server="true" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808271 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808275 2564 flags.go:64] FLAG: --event-burst="100" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808278 2564 flags.go:64] FLAG: --event-qps="50" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808281 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808284 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808287 2564 flags.go:64] FLAG: --eviction-hard="" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808291 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808293 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808296 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808299 2564 flags.go:64] FLAG: --eviction-soft="" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808302 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808305 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808308 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808311 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808314 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808317 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808320 2564 flags.go:64] FLAG: --feature-gates="" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808323 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 00:23:27.878502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808328 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808332 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808335 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808338 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808341 2564 flags.go:64] FLAG: --help="false" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808344 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-139-1.ec2.internal" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808347 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808350 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808353 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808358 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808361 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808364 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808367 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808370 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808373 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808376 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808379 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808382 2564 flags.go:64] FLAG: --kube-reserved="" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808385 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808388 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808391 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808394 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808397 2564 flags.go:64] FLAG: --lock-file="" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808399 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 00:23:27.879655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808402 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808405 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808410 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808413 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808415 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808418 2564 flags.go:64] FLAG: --logging-format="text" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808421 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808424 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808428 2564 flags.go:64] FLAG: --manifest-url="" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808431 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808435 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808438 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808444 2564 flags.go:64] FLAG: --max-pods="110" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808447 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808450 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808453 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808456 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808460 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808463 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808466 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808473 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808476 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808479 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 00:23:28.014298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808482 2564 flags.go:64] FLAG: --pod-cidr="" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808488 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808493 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808496 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808499 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808502 2564 flags.go:64] FLAG: --port="10250" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808505 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808508 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e73844f1890300f9" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808511 2564 flags.go:64] FLAG: --qos-reserved="" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808514 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808517 2564 flags.go:64] FLAG: --register-node="true" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808520 2564 flags.go:64] FLAG: --register-with-taints="" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808542 2564 flags.go:64] FLAG: --registry-burst="10" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808545 2564 flags.go:64] FLAG: --registry-qps="5" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808548 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808551 2564 flags.go:64] FLAG: --reserved-memory="" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808555 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808558 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808562 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808565 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808568 2564 flags.go:64] FLAG: --runonce="false" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808571 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808574 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808577 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808580 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 00:23:28.029897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808583 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808587 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808590 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808593 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808596 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808599 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808602 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808605 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808608 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808611 2564 flags.go:64] FLAG: --system-cgroups="" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808614 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808620 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808622 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808625 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808630 2564 flags.go:64] FLAG: --tls-min-version="" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808633 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808635 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808639 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808641 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808644 2564 flags.go:64] FLAG: --v="2" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808648 2564 flags.go:64] FLAG: --version="false" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808652 2564 flags.go:64] FLAG: --vmodule="" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808655 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.808659 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810021 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:23:28.033352 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810037 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810043 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810048 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810054 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810058 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810063 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810068 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810072 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810076 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810080 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810085 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810088 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810091 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810094 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810096 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810099 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810102 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810105 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810108 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810110 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:23:28.034244 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810113 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810116 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810118 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810120 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810123 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810126 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810128 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810131 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810135 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810140 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810147 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810151 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810158 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810162 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810165 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810167 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810170 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810173 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810175 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:23:28.034878 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810178 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810181 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810183 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810186 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810189 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810193 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810205 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810209 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810214 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810217 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810222 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810226 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810230 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810234 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810237 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810240 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810244 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810250 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810255 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810259 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:23:28.035495 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810264 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810267 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810270 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810274 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810278 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810283 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810287 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810292 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810295 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810299 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810303 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810307 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810311 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810314 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810318 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810322 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810326 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810334 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810338 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810343 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:23:28.036090 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810346 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810350 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810354 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810358 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810362 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810366 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810369 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810373 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810376 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810380 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810384 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810387 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810391 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810395 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810399 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810404 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810407 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810412 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:23:28.036718 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.810416 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:23:28.037201 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.811114 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 00:23:28.037201 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.818053 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 00:23:28.037201 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.818068 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 00:23:28.037201 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.818087 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 00:23:28.037201 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.818093 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 00:23:28.037201 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818145 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:23:28.037201 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818150 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:23:28.037201 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818153 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:23:28.037201 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818157 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:23:28.037201 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818160 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:23:28.037201 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818163 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:23:28.037201 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818167 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:23:28.037201 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818169 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:23:28.037201 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818172 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818174 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818177 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818180 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818182 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818185 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818188 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818191 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818193 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818221 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818226 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818229 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818231 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818234 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818237 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818240 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818243 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818245 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818248 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818250 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:23:28.037667 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818253 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818260 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818263 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818266 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818269 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818271 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818274 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818276 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818279 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818281 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818284 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818286 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818289 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818292 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818295 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818298 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818301 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818303 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818306 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818308 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:23:28.038214 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818311 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818315 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818317 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818321 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818323 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818326 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818329 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818331 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818334 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818336 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818340 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818343 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818345 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818348 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818351 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818355 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818357 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818360 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818362 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818364 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:23:28.038880 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818367 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818370 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818373 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818375 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818377 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818380 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818384 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818388 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818391 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818394 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818396 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818399 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818402 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818407 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818411 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818415 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818418 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818422 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818425 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:23:28.039418 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818427 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818430 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818433 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818437 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818440 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818443 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818446 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818448 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818451 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818454 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818457 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818459 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.818464 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818584 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818590 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818593 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 00:23:28.207619 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818596 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818598 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818603 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818606 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818610 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818613 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818616 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818619 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818622 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818625 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818628 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818632 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818635 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818637 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818640 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818643 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818645 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818648 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818650 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 00:23:28.208466 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818654 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818657 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818660 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818662 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818665 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818667 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818674 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818677 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818680 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818682 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818685 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818687 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818690 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818692 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818695 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818698 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818700 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818703 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818705 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 00:23:28.209110 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818708 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818710 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818713 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818715 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818718 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818721 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818724 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818726 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818729 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818731 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818734 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818737 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818740 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818743 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818746 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818748 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818751 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818754 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818756 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818759 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 00:23:28.209820 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818762 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818764 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818767 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818770 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818773 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818776 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818778 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818781 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818783 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818786 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818788 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818791 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818795 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818798 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818801 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818803 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818806 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818810 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818813 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 00:23:28.210514 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818815 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818817 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818820 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818823 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818825 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818828 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818832 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818835 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818837 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818840 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818843 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818846 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818848 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818851 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818854 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818857 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818860 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818862 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 00:23:28.211235 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:27.818865 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 00:23:28.211982 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.818870 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 00:23:28.211982 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.818969 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 00:23:28.211982 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.823122 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 00:23:28.211982 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.824492 2564 server.go:1019] "Starting client certificate rotation" Jul 23 00:23:28.211982 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.824594 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 00:23:28.211982 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.824624 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 00:23:28.211982 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.826946 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 00:23:28.211982 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.828077 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 00:23:28.211982 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.839188 2564 log.go:25] "Validated CRI v1 runtime API" Jul 23 00:23:28.211982 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.842711 2564 log.go:25] "Validated CRI v1 image API" Jul 23 00:23:28.211982 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.843376 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 00:23:28.211982 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.845976 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7abf27e5-ac41-4810-8439-1486c1e3bb18:/dev/nvme0n1p4 c481be20-11e5-4644-813b-07ec09ab168c:/dev/nvme0n1p3] Jul 23 00:23:28.212643 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.845997 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 00:23:28.212858 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.850436 2564 manager.go:216] Machine: {Timestamp:2026-07-23 00:23:27.849960466 +0000 UTC m=+0.345629170 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100870 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2242eabda88653fad2d7a8d07b00be SystemUUID:ec2242ea-bda8-8653-fad2-d7a8d07b00be BootID:de0c1cd6-7e6d-46e3-9d98-ffd20f54f77f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:44:b9:33:69:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:44:b9:33:69:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:04:6e:a5:fd:ee Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 00:23:28.212858 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.850547 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 00:23:28.212858 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.850648 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 00:23:28.212858 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.853667 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 00:23:28.212858 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.853689 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-1.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 00:23:28.212858 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.854471 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 00:23:28.212858 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.854481 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 00:23:28.212858 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.854489 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.854499 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.855722 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.858650 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.858777 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.859565 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.859580 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.859598 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.859610 2564 kubelet.go:398] "Adding apiserver pod source" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.859625 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.860025 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.860041 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.862111 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.862346 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.864120 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.864148 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.865410 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.865423 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.865429 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.865436 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.865444 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.865461 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.865469 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.865475 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.865481 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.865487 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.865499 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.865504 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.866308 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.866316 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:27.868878 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:27.868879 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-1.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.869387 2564 server.go:1268] "Started kubelet" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.869513 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.869500 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.869621 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.871661 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.873499 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:27.876327 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 00:23:28.215736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.878779 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-1.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.879068 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.879155 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:27.879253 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-1.ec2.internal\" not found" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.880335 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.880342 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.880386 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:27.880804 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-1.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.881044 2564 factory.go:153] Registering CRI-O factory Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.881060 2564 factory.go:223] Registration of the crio container factory successfully Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.881120 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.881131 2564 factory.go:55] Registering systemd factory Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.881142 2564 factory.go:223] Registration of the systemd container factory successfully Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.881167 2564 factory.go:103] Registering Raw factory Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.881201 2564 manager.go:1196] Started watching for new ooms in manager Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.881630 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.881701 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.882064 2564 manager.go:349] Starting recovery of all containers Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.888078 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cbx7h" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:27.878547 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-1.ec2.internal.18c4c368f32cb789 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-1.ec2.internal,UID:ip-10-0-139-1.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-1.ec2.internal,},FirstTimestamp:2026-07-23 00:23:27.869368201 +0000 UTC m=+0.365036905,LastTimestamp:2026-07-23 00:23:27.869368201 +0000 UTC m=+0.365036905,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-1.ec2.internal,}" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.890559 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cbx7h" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:27.895006 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.899200 2564 manager.go:354] Recovery completed Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.904116 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.905829 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.905852 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.905865 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.906513 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.906538 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.906555 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.906806 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.907450 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.907467 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:23:28.217520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.907478 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.907690 2564 policy_none.go:49] "None policy: Start" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.907704 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.907716 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.908742 2564 policy_none.go:47] "Start" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.944075 2564 manager.go:337] "Starting Device Plugin manager" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:27.944220 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.944232 2564 server.go:87] "Starting device plugin registration server" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.944454 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.944466 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.944587 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.944666 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:27.944677 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:27.945143 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:27.945197 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-1.ec2.internal\" not found" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.044704 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.045513 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.045552 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.045570 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.045592 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.051208 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:28.051225 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-1.ec2.internal\": node \"ip-10-0-139-1.ec2.internal\" not found" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.061817 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.066157 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.066190 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.066215 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:28.066252 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.068693 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:28.070588 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-1.ec2.internal\" not found" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.167869 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-1.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal"] Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.167932 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.169858 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.169883 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.169893 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:28.170970 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-1.ec2.internal\" not found" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.171156 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.171263 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.171295 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.172471 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:23:28.377889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.172476 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.172494 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.172497 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.172506 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.172507 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.174035 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.174057 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.174776 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasSufficientMemory" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.174804 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.174819 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeHasSufficientPID" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:28.206281 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-1.ec2.internal\" not found" node="ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:28.219590 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-1.ec2.internal\" not found" node="ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:28.271705 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-1.ec2.internal\" not found" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.285567 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4359bec0e352431350db386bdadf6398-config\") pod \"kube-apiserver-proxy-ip-10-0-139-1.ec2.internal\" (UID: \"4359bec0e352431350db386bdadf6398\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.285598 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a315493ff7ec57552d98ccbaa050d33e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal\" (UID: \"a315493ff7ec57552d98ccbaa050d33e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.285615 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a315493ff7ec57552d98ccbaa050d33e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal\" (UID: \"a315493ff7ec57552d98ccbaa050d33e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.379188 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:28.372744 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-1.ec2.internal\" not found" Jul 23 00:23:28.588729 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.385957 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a315493ff7ec57552d98ccbaa050d33e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal\" (UID: \"a315493ff7ec57552d98ccbaa050d33e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.588729 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.386014 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a315493ff7ec57552d98ccbaa050d33e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal\" (UID: \"a315493ff7ec57552d98ccbaa050d33e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.588729 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.386066 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a315493ff7ec57552d98ccbaa050d33e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal\" (UID: \"a315493ff7ec57552d98ccbaa050d33e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.588729 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.386096 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4359bec0e352431350db386bdadf6398-config\") pod \"kube-apiserver-proxy-ip-10-0-139-1.ec2.internal\" (UID: \"4359bec0e352431350db386bdadf6398\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.588729 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.386134 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4359bec0e352431350db386bdadf6398-config\") pod \"kube-apiserver-proxy-ip-10-0-139-1.ec2.internal\" (UID: \"4359bec0e352431350db386bdadf6398\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.588729 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.386149 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a315493ff7ec57552d98ccbaa050d33e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal\" (UID: \"a315493ff7ec57552d98ccbaa050d33e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.588729 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:28.479216 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-1.ec2.internal\" not found" Jul 23 00:23:28.588729 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.509380 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.588729 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.522885 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal" Jul 23 00:23:28.588729 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:28.579872 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-1.ec2.internal\" not found" Jul 23 00:23:28.769747 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:28.687015 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-1.ec2.internal\" not found" Jul 23 00:23:28.835817 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:28.787229 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-1.ec2.internal\" not found" Jul 23 00:23:28.835817 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.824135 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 00:23:28.835817 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.824280 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 00:23:29.030028 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:28.890371 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-1.ec2.internal\" not found" Jul 23 00:23:29.030028 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.895005 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 00:18:28 +0000 UTC" deadline="2028-05-07 11:38:59.695916782 +0000 UTC" Jul 23 00:23:29.030028 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.895034 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15707h15m30.800885601s" Jul 23 00:23:29.030028 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.927261 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:23:29.030028 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.979514 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-1.ec2.internal" Jul 23 00:23:29.030028 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.990354 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 00:23:29.030028 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:28.992431 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal" Jul 23 00:23:29.030028 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.004187 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:23:29.030028 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.004668 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 00:23:29.265389 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:29.220065 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda315493ff7ec57552d98ccbaa050d33e.slice/crio-e50d3fc003e4dd4c4563b221fa2a4c6ea7a6803396bcc337fc2dc5a2ece2308c WatchSource:0}: Error finding container e50d3fc003e4dd4c4563b221fa2a4c6ea7a6803396bcc337fc2dc5a2ece2308c: Status 404 returned error can't find the container with id e50d3fc003e4dd4c4563b221fa2a4c6ea7a6803396bcc337fc2dc5a2ece2308c Jul 23 00:23:29.265389 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:29.220493 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4359bec0e352431350db386bdadf6398.slice/crio-c99b2540ebea1fabd82e8842d0538631e69481b788807184683f0e5ac8b018fa WatchSource:0}: Error finding container c99b2540ebea1fabd82e8842d0538631e69481b788807184683f0e5ac8b018fa: Status 404 returned error can't find the container with id c99b2540ebea1fabd82e8842d0538631e69481b788807184683f0e5ac8b018fa Jul 23 00:23:29.265389 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.225265 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:23:29.425065 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.338782 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:23:29.886470 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.860836 2564 apiserver.go:52] "Watching apiserver" Jul 23 00:23:29.886470 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.869868 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 00:23:29.886470 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.871617 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-1.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh","openshift-image-registry/node-ca-zxnfn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal","openshift-multus/network-metrics-daemon-mx6kd","openshift-network-operator/iptables-alerter-xqsl8","openshift-ovn-kubernetes/ovnkube-node-hq5hv","kube-system/global-pull-secret-syncer-njhfs","kube-system/konnectivity-agent-82f2h","openshift-cluster-node-tuning-operator/tuned-dxrmh","openshift-dns/node-resolver-jl4hg","openshift-multus/multus-additional-cni-plugins-h6c6l","openshift-multus/multus-tf94d","openshift-network-diagnostics/network-check-target-vvs7d"] Jul 23 00:23:29.886470 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.873835 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:29.886470 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.876617 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 00:23:29.886470 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.877154 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-29frj\"" Jul 23 00:23:29.886470 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.877359 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 00:23:29.886470 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.877527 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 00:23:29.886470 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.878375 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 00:23:29.886470 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.878651 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 00:23:29.886470 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.880021 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 00:23:29.886470 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.886086 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 00:23:29.887163 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.886494 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-82f2h" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.889859 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.889936 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:29.890057 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.890816 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-fxg4c\"" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.890965 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.892576 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.895196 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xqsl8" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.913554 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.918705 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-54zb7\"" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.919514 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-run-ovn\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.919563 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sb5vl" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.919567 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-node-log\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.919655 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-log-socket\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.919724 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-systemd-units\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.919789 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-run-netns\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.919835 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-etc-openvswitch\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.919882 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4123100-2f7f-4b30-834f-04d2de11303b-ovnkube-config\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.919931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6411344d-2ba4-418f-9b2c-e5cd631325b6-agent-certs\") pod \"konnectivity-agent-82f2h\" (UID: \"6411344d-2ba4-418f-9b2c-e5cd631325b6\") " pod="kube-system/konnectivity-agent-82f2h" Jul 23 00:23:29.958212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.919976 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-slash\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.037113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.920016 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-var-lib-openvswitch\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.037113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.920068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.037113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.920110 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-cni-bin\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.037113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.920155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4123100-2f7f-4b30-834f-04d2de11303b-env-overrides\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.037113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.920190 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4123100-2f7f-4b30-834f-04d2de11303b-ovnkube-script-lib\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.037113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.920218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjjx\" (UniqueName: \"kubernetes.io/projected/e4123100-2f7f-4b30-834f-04d2de11303b-kube-api-access-msjjx\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.037113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.920272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-kubelet\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.037113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.920305 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-run-systemd\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.037113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.920348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-run-openvswitch\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.037113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.920382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-cni-netd\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.037113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.920453 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zxnfn" Jul 23 00:23:30.037113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.920465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.037113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.920670 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 00:23:30.037113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.920771 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4123100-2f7f-4b30-834f-04d2de11303b-ovn-node-metrics-cert\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.037113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.920809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6411344d-2ba4-418f-9b2c-e5cd631325b6-konnectivity-ca\") pod \"konnectivity-agent-82f2h\" (UID: \"6411344d-2ba4-418f-9b2c-e5cd631325b6\") " pod="kube-system/konnectivity-agent-82f2h" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.924262 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.925439 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.925474 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2kxk6\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.925515 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sb5vl" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.925655 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.926266 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:29.926337 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.927651 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.928288 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.930435 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.931127 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.931142 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.931400 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dcx9m\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.931592 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.932137 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jl4hg" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.933087 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6zsrf\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.933126 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.933241 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.933839 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.935427 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.937025 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.937338 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:29.937438 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.937828 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.937842 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.937974 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.938021 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 00:23:30.038664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.938277 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.938361 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cgck4\"" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.938383 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.938468 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fwwbr\"" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.938523 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cghjk\"" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.938553 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:29.981815 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039368 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-cnibin\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039410 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-kubelet\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-run-k8s-cni-cncf-io\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039472 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-var-lib-cni-bin\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-multus-conf-dir\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039577 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-kubelet\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039582 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-socket-dir\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-registration-dir\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-var-lib-kubelet\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-node-log\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039816 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-node-log\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.039951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-systemd-units\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.044642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039910 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.044642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039939 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-etc-selinux\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.044642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039954 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-systemd-units\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.044642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-sys-fs\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.044642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.039998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-host\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.044642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-multus-cni-dir\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.044642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040066 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-sys\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.044642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4123100-2f7f-4b30-834f-04d2de11303b-env-overrides\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.044642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4123100-2f7f-4b30-834f-04d2de11303b-ovnkube-script-lib\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.044642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n492\" (UniqueName: \"kubernetes.io/projected/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-kube-api-access-7n492\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.044642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040188 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-sysconfig\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.044642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040215 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3e1769c-38a9-4493-8e79-c758ea29b40a-cnibin\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.044642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3e1769c-38a9-4493-8e79-c758ea29b40a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.044642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040273 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb7bl\" (UniqueName: \"kubernetes.io/projected/c3e1769c-38a9-4493-8e79-c758ea29b40a-kube-api-access-bb7bl\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.045777 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040306 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-run-systemd\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.045777 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040344 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-run-openvswitch\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.045777 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040391 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6411344d-2ba4-418f-9b2c-e5cd631325b6-konnectivity-ca\") pod \"konnectivity-agent-82f2h\" (UID: \"6411344d-2ba4-418f-9b2c-e5cd631325b6\") " pod="kube-system/konnectivity-agent-82f2h" Jul 23 00:23:30.045777 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040417 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgb8t\" (UniqueName: \"kubernetes.io/projected/c0e82032-5917-427c-b7b6-e7597a1390d3-kube-api-access-mgb8t\") pod \"node-ca-zxnfn\" (UID: \"c0e82032-5917-427c-b7b6-e7597a1390d3\") " pod="openshift-image-registry/node-ca-zxnfn" Jul 23 00:23:30.045777 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040432 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-run-systemd\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.045777 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040483 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-run-openvswitch\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.045777 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040531 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-systemd\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.045777 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040744 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3e1769c-38a9-4493-8e79-c758ea29b40a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.045777 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqsct\" (UniqueName: \"kubernetes.io/projected/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-kube-api-access-sqsct\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.045777 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040779 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4123100-2f7f-4b30-834f-04d2de11303b-env-overrides\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.045777 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-run-netns\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.045777 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4123100-2f7f-4b30-834f-04d2de11303b-ovnkube-config\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.045777 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3e1769c-38a9-4493-8e79-c758ea29b40a-system-cni-dir\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.045777 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjjx\" (UniqueName: \"kubernetes.io/projected/e4123100-2f7f-4b30-834f-04d2de11303b-kube-api-access-msjjx\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.047437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040925 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/75dffbc4-21a5-4051-8cae-f7f9988f6630-kubelet-config\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:30.047437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040948 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-run-netns\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.047437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040969 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-device-dir\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.047437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.040992 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91698027-ce8c-4729-86e9-6eb62632951d-tmp-dir\") pod \"node-resolver-jl4hg\" (UID: \"91698027-ce8c-4729-86e9-6eb62632951d\") " pod="openshift-dns/node-resolver-jl4hg" Jul 23 00:23:30.047437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041011 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc61300d-11be-4c5d-a1fd-6e305d4e193a-tmp\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.047437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041012 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4123100-2f7f-4b30-834f-04d2de11303b-ovnkube-script-lib\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.047437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3e1769c-38a9-4493-8e79-c758ea29b40a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.047437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-cni-binary-copy\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.047437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041112 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6411344d-2ba4-418f-9b2c-e5cd631325b6-konnectivity-ca\") pod \"konnectivity-agent-82f2h\" (UID: \"6411344d-2ba4-418f-9b2c-e5cd631325b6\") " pod="kube-system/konnectivity-agent-82f2h" Jul 23 00:23:30.047437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041239 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-run-netns\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.047437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-cni-netd\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.047437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041359 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.047437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041406 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.047437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4123100-2f7f-4b30-834f-04d2de11303b-ovn-node-metrics-cert\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.048741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041447 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-cni-netd\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.048741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041469 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4123100-2f7f-4b30-834f-04d2de11303b-ovnkube-config\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.048741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041501 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0e82032-5917-427c-b7b6-e7597a1390d3-host\") pod \"node-ca-zxnfn\" (UID: \"c0e82032-5917-427c-b7b6-e7597a1390d3\") " pod="openshift-image-registry/node-ca-zxnfn" Jul 23 00:23:30.048741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041536 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3e1769c-38a9-4493-8e79-c758ea29b40a-os-release\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.048741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041561 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb86z\" (UniqueName: \"kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z\") pod \"network-check-target-vvs7d\" (UID: \"85661880-f744-4f0a-a606-822e19bba144\") " pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:30.048741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041588 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-run-ovn\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.048741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-sysctl-conf\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.048741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041682 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3e1769c-38a9-4493-8e79-c758ea29b40a-cni-binary-copy\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.048741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041729 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-run-ovn\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.048741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041746 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 00:23:30.048741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041777 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-var-lib-kubelet\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.048741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041806 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-multus-daemon-config\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.048741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041837 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-etc-openvswitch\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.048741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041882 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-etc-openvswitch\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.050373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041936 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a62da51e-b639-41a0-b6dc-0d48da42b3c5-host-slash\") pod \"iptables-alerter-xqsl8\" (UID: \"a62da51e-b639-41a0-b6dc-0d48da42b3c5\") " pod="openshift-network-operator/iptables-alerter-xqsl8" Jul 23 00:23:30.050373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.041989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdgmw\" (UniqueName: \"kubernetes.io/projected/a62da51e-b639-41a0-b6dc-0d48da42b3c5-kube-api-access-mdgmw\") pod \"iptables-alerter-xqsl8\" (UID: \"a62da51e-b639-41a0-b6dc-0d48da42b3c5\") " pod="openshift-network-operator/iptables-alerter-xqsl8" Jul 23 00:23:30.050373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042015 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-system-cni-dir\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.050373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-os-release\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.050373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-sysctl-d\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.050373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042103 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-slash\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.050373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042139 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-var-lib-openvswitch\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.050373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042162 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a62da51e-b639-41a0-b6dc-0d48da42b3c5-iptables-alerter-script\") pod \"iptables-alerter-xqsl8\" (UID: \"a62da51e-b639-41a0-b6dc-0d48da42b3c5\") " pod="openshift-network-operator/iptables-alerter-xqsl8" Jul 23 00:23:30.050373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042203 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6vhf\" (UniqueName: \"kubernetes.io/projected/dc61300d-11be-4c5d-a1fd-6e305d4e193a-kube-api-access-r6vhf\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.050373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-var-lib-openvswitch\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.050373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs\") pod \"network-metrics-daemon-mx6kd\" (UID: \"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246\") " pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:30.050373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042250 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:30.050373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042200 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-slash\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.050373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042297 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91698027-ce8c-4729-86e9-6eb62632951d-hosts-file\") pod \"node-resolver-jl4hg\" (UID: \"91698027-ce8c-4729-86e9-6eb62632951d\") " pod="openshift-dns/node-resolver-jl4hg" Jul 23 00:23:30.052248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsjpc\" (UniqueName: \"kubernetes.io/projected/91698027-ce8c-4729-86e9-6eb62632951d-kube-api-access-qsjpc\") pod \"node-resolver-jl4hg\" (UID: \"91698027-ce8c-4729-86e9-6eb62632951d\") " pod="openshift-dns/node-resolver-jl4hg" Jul 23 00:23:30.052248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042373 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-multus-socket-dir-parent\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.052248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042404 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-run-multus-certs\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.052248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042638 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr65w\" (UniqueName: \"kubernetes.io/projected/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-kube-api-access-lr65w\") pod \"network-metrics-daemon-mx6kd\" (UID: \"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246\") " pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:30.052248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042722 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0e82032-5917-427c-b7b6-e7597a1390d3-serviceca\") pod \"node-ca-zxnfn\" (UID: \"c0e82032-5917-427c-b7b6-e7597a1390d3\") " pod="openshift-image-registry/node-ca-zxnfn" Jul 23 00:23:30.052248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042755 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-kubernetes\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.052248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042794 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-run\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.052248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042836 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-hostroot\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.052248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042858 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-etc-kubernetes\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.052248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-log-socket\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.052248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6411344d-2ba4-418f-9b2c-e5cd631325b6-agent-certs\") pod \"konnectivity-agent-82f2h\" (UID: \"6411344d-2ba4-418f-9b2c-e5cd631325b6\") " pod="kube-system/konnectivity-agent-82f2h" Jul 23 00:23:30.052248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.042950 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-lib-modules\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.052248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.043003 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-tuned\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.052248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.043027 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-var-lib-cni-multus\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.052893 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.043056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.052893 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.043089 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-cni-bin\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.052893 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.043112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/75dffbc4-21a5-4051-8cae-f7f9988f6630-dbus\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:30.052893 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.043134 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-modprobe-d\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.052893 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.043294 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-log-socket\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.052893 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.045700 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4123100-2f7f-4b30-834f-04d2de11303b-ovn-node-metrics-cert\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.052893 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.045777 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.052893 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.045857 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4123100-2f7f-4b30-834f-04d2de11303b-host-cni-bin\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.052893 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.049315 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6411344d-2ba4-418f-9b2c-e5cd631325b6-agent-certs\") pod \"konnectivity-agent-82f2h\" (UID: \"6411344d-2ba4-418f-9b2c-e5cd631325b6\") " pod="kube-system/konnectivity-agent-82f2h" Jul 23 00:23:30.052893 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.052061 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjjx\" (UniqueName: \"kubernetes.io/projected/e4123100-2f7f-4b30-834f-04d2de11303b-kube-api-access-msjjx\") pod \"ovnkube-node-hq5hv\" (UID: \"e4123100-2f7f-4b30-834f-04d2de11303b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.072207 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.072162 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-1.ec2.internal" event={"ID":"4359bec0e352431350db386bdadf6398","Type":"ContainerStarted","Data":"c99b2540ebea1fabd82e8842d0538631e69481b788807184683f0e5ac8b018fa"} Jul 23 00:23:30.073851 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.073826 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal" event={"ID":"a315493ff7ec57552d98ccbaa050d33e","Type":"ContainerStarted","Data":"e50d3fc003e4dd4c4563b221fa2a4c6ea7a6803396bcc337fc2dc5a2ece2308c"} Jul 23 00:23:30.143815 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.143782 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n492\" (UniqueName: \"kubernetes.io/projected/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-kube-api-access-7n492\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.143919 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.143822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-sysconfig\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.143919 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.143854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3e1769c-38a9-4493-8e79-c758ea29b40a-cnibin\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.143919 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.143909 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3e1769c-38a9-4493-8e79-c758ea29b40a-cnibin\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.143919 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.143914 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-sysconfig\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.144090 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.143948 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3e1769c-38a9-4493-8e79-c758ea29b40a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.144090 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.143980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7bl\" (UniqueName: \"kubernetes.io/projected/c3e1769c-38a9-4493-8e79-c758ea29b40a-kube-api-access-bb7bl\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.144090 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgb8t\" (UniqueName: \"kubernetes.io/projected/c0e82032-5917-427c-b7b6-e7597a1390d3-kube-api-access-mgb8t\") pod \"node-ca-zxnfn\" (UID: \"c0e82032-5917-427c-b7b6-e7597a1390d3\") " pod="openshift-image-registry/node-ca-zxnfn" Jul 23 00:23:30.144090 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-systemd\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.144090 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3e1769c-38a9-4493-8e79-c758ea29b40a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.144090 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqsct\" (UniqueName: \"kubernetes.io/projected/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-kube-api-access-sqsct\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.144314 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3e1769c-38a9-4493-8e79-c758ea29b40a-system-cni-dir\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.144314 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144142 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/75dffbc4-21a5-4051-8cae-f7f9988f6630-kubelet-config\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:30.144314 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144156 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3e1769c-38a9-4493-8e79-c758ea29b40a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.144314 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-device-dir\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.144314 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144212 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91698027-ce8c-4729-86e9-6eb62632951d-tmp-dir\") pod \"node-resolver-jl4hg\" (UID: \"91698027-ce8c-4729-86e9-6eb62632951d\") " pod="openshift-dns/node-resolver-jl4hg" Jul 23 00:23:30.144314 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144236 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc61300d-11be-4c5d-a1fd-6e305d4e193a-tmp\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.144314 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144241 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-systemd\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.144314 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144279 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3e1769c-38a9-4493-8e79-c758ea29b40a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.144314 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-cni-binary-copy\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.144314 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144313 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3e1769c-38a9-4493-8e79-c758ea29b40a-system-cni-dir\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.144808 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-run-netns\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.144808 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144371 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0e82032-5917-427c-b7b6-e7597a1390d3-host\") pod \"node-ca-zxnfn\" (UID: \"c0e82032-5917-427c-b7b6-e7597a1390d3\") " pod="openshift-image-registry/node-ca-zxnfn" Jul 23 00:23:30.144808 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3e1769c-38a9-4493-8e79-c758ea29b40a-os-release\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.144808 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144435 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb86z\" (UniqueName: \"kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z\") pod \"network-check-target-vvs7d\" (UID: \"85661880-f744-4f0a-a606-822e19bba144\") " pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:30.144808 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-sysctl-conf\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.144808 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3e1769c-38a9-4493-8e79-c758ea29b40a-cni-binary-copy\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.144808 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-var-lib-kubelet\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.144808 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144551 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-multus-daemon-config\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.144808 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a62da51e-b639-41a0-b6dc-0d48da42b3c5-host-slash\") pod \"iptables-alerter-xqsl8\" (UID: \"a62da51e-b639-41a0-b6dc-0d48da42b3c5\") " pod="openshift-network-operator/iptables-alerter-xqsl8" Jul 23 00:23:30.144808 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdgmw\" (UniqueName: \"kubernetes.io/projected/a62da51e-b639-41a0-b6dc-0d48da42b3c5-kube-api-access-mdgmw\") pod \"iptables-alerter-xqsl8\" (UID: \"a62da51e-b639-41a0-b6dc-0d48da42b3c5\") " pod="openshift-network-operator/iptables-alerter-xqsl8" Jul 23 00:23:30.144808 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144659 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-system-cni-dir\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.144808 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-os-release\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.144808 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144683 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/75dffbc4-21a5-4051-8cae-f7f9988f6630-kubelet-config\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:30.144808 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-sysctl-d\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.145298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a62da51e-b639-41a0-b6dc-0d48da42b3c5-iptables-alerter-script\") pod \"iptables-alerter-xqsl8\" (UID: \"a62da51e-b639-41a0-b6dc-0d48da42b3c5\") " pod="openshift-network-operator/iptables-alerter-xqsl8" Jul 23 00:23:30.145298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144751 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-device-dir\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.145298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6vhf\" (UniqueName: \"kubernetes.io/projected/dc61300d-11be-4c5d-a1fd-6e305d4e193a-kube-api-access-r6vhf\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.145298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs\") pod \"network-metrics-daemon-mx6kd\" (UID: \"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246\") " pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:30.145298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144809 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-var-lib-kubelet\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.145298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:30.145298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91698027-ce8c-4729-86e9-6eb62632951d-hosts-file\") pod \"node-resolver-jl4hg\" (UID: \"91698027-ce8c-4729-86e9-6eb62632951d\") " pod="openshift-dns/node-resolver-jl4hg" Jul 23 00:23:30.145298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsjpc\" (UniqueName: \"kubernetes.io/projected/91698027-ce8c-4729-86e9-6eb62632951d-kube-api-access-qsjpc\") pod \"node-resolver-jl4hg\" (UID: \"91698027-ce8c-4729-86e9-6eb62632951d\") " pod="openshift-dns/node-resolver-jl4hg" Jul 23 00:23:30.145298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144907 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-multus-socket-dir-parent\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.145298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144927 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-cni-binary-copy\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.145298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144932 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-run-multus-certs\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.145298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144959 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr65w\" (UniqueName: \"kubernetes.io/projected/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-kube-api-access-lr65w\") pod \"network-metrics-daemon-mx6kd\" (UID: \"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246\") " pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:30.145298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.144985 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0e82032-5917-427c-b7b6-e7597a1390d3-serviceca\") pod \"node-ca-zxnfn\" (UID: \"c0e82032-5917-427c-b7b6-e7597a1390d3\") " pod="openshift-image-registry/node-ca-zxnfn" Jul 23 00:23:30.145298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-kubernetes\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.145846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145043 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-run\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.145846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-hostroot\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.145846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145102 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-etc-kubernetes\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.145846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145090 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91698027-ce8c-4729-86e9-6eb62632951d-tmp-dir\") pod \"node-resolver-jl4hg\" (UID: \"91698027-ce8c-4729-86e9-6eb62632951d\") " pod="openshift-dns/node-resolver-jl4hg" Jul 23 00:23:30.145846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145161 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-etc-kubernetes\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.145846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145181 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-lib-modules\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.145846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-tuned\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.145846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145242 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-var-lib-cni-multus\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.145846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145274 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/75dffbc4-21a5-4051-8cae-f7f9988f6630-dbus\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:30.145846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145302 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-modprobe-d\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.145846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145329 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-cnibin\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.145846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145360 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-run-k8s-cni-cncf-io\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.145846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145387 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-var-lib-cni-bin\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.145846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145414 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-multus-conf-dir\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.145846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-socket-dir\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.146351 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-registration-dir\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.146351 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145501 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c3e1769c-38a9-4493-8e79-c758ea29b40a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.146351 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145507 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c3e1769c-38a9-4493-8e79-c758ea29b40a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.146351 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145562 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91698027-ce8c-4729-86e9-6eb62632951d-hosts-file\") pod \"node-resolver-jl4hg\" (UID: \"91698027-ce8c-4729-86e9-6eb62632951d\") " pod="openshift-dns/node-resolver-jl4hg" Jul 23 00:23:30.146351 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145564 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0e82032-5917-427c-b7b6-e7597a1390d3-host\") pod \"node-ca-zxnfn\" (UID: \"c0e82032-5917-427c-b7b6-e7597a1390d3\") " pod="openshift-image-registry/node-ca-zxnfn" Jul 23 00:23:30.146351 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145575 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-var-lib-kubelet\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.146351 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145616 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a62da51e-b639-41a0-b6dc-0d48da42b3c5-host-slash\") pod \"iptables-alerter-xqsl8\" (UID: \"a62da51e-b639-41a0-b6dc-0d48da42b3c5\") " pod="openshift-network-operator/iptables-alerter-xqsl8" Jul 23 00:23:30.146351 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145640 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-run-netns\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.146351 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-var-lib-kubelet\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.146351 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145655 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3e1769c-38a9-4493-8e79-c758ea29b40a-os-release\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.146351 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145661 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-multus-daemon-config\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.146351 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.146351 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-etc-selinux\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.146351 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-sys-fs\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.147014 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-host\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.147014 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145809 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-system-cni-dir\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.147014 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-multus-cni-dir\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.147014 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-sys\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.147014 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145856 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-os-release\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.147014 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145926 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-host\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.147014 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.145969 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-sysctl-conf\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.147014 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146054 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-multus-cni-dir\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.147014 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146068 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-cnibin\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.147014 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146047 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-multus-socket-dir-parent\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.147014 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146113 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-sys\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.147014 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146116 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-run-k8s-cni-cncf-io\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.147014 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146147 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-modprobe-d\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.147014 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146159 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-var-lib-cni-bin\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.147014 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146196 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-sysctl-d\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.147892 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146228 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-registration-dir\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.147892 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146247 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-kubernetes\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.147892 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146266 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-run-multus-certs\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.147892 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146284 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3e1769c-38a9-4493-8e79-c758ea29b40a-cni-binary-copy\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.147892 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146376 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-lib-modules\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.147892 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146724 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-socket-dir\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.147892 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146802 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a62da51e-b639-41a0-b6dc-0d48da42b3c5-iptables-alerter-script\") pod \"iptables-alerter-xqsl8\" (UID: \"a62da51e-b639-41a0-b6dc-0d48da42b3c5\") " pod="openshift-network-operator/iptables-alerter-xqsl8" Jul 23 00:23:30.147892 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.146882 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:23:30.147892 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146889 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-host-var-lib-cni-multus\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.147892 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146889 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0e82032-5917-427c-b7b6-e7597a1390d3-serviceca\") pod \"node-ca-zxnfn\" (UID: \"c0e82032-5917-427c-b7b6-e7597a1390d3\") " pod="openshift-image-registry/node-ca-zxnfn" Jul 23 00:23:30.147892 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146933 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-sys-fs\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.147892 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.146948 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs podName:9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246 nodeName:}" failed. No retries permitted until 2026-07-23 00:23:30.646927244 +0000 UTC m=+3.142595956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs") pod "network-metrics-daemon-mx6kd" (UID: "9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:23:30.147892 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146944 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.147892 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.146987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-etc-selinux\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.148753 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.147001 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-multus-conf-dir\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.148753 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.147003 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-hostroot\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.148753 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.147039 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:23:30.148753 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.147031 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dc61300d-11be-4c5d-a1fd-6e305d4e193a-run\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.148753 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.147099 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/75dffbc4-21a5-4051-8cae-f7f9988f6630-dbus\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:30.148753 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.147155 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret podName:75dffbc4-21a5-4051-8cae-f7f9988f6630 nodeName:}" failed. No retries permitted until 2026-07-23 00:23:30.647141075 +0000 UTC m=+3.142809772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret") pod "global-pull-secret-syncer-njhfs" (UID: "75dffbc4-21a5-4051-8cae-f7f9988f6630") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:23:30.148753 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.147230 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc61300d-11be-4c5d-a1fd-6e305d4e193a-tmp\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.149121 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.148808 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dc61300d-11be-4c5d-a1fd-6e305d4e193a-etc-tuned\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.152273 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.151864 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:23:30.152273 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.151888 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:23:30.152273 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.151913 2564 projected.go:196] Error preparing data for projected volume kube-api-access-hb86z for pod openshift-network-diagnostics/network-check-target-vvs7d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:23:30.152273 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.151996 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z podName:85661880-f744-4f0a-a606-822e19bba144 nodeName:}" failed. No retries permitted until 2026-07-23 00:23:30.651963241 +0000 UTC m=+3.147631933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hb86z" (UniqueName: "kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z") pod "network-check-target-vvs7d" (UID: "85661880-f744-4f0a-a606-822e19bba144") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:23:30.153768 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.153739 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgb8t\" (UniqueName: \"kubernetes.io/projected/c0e82032-5917-427c-b7b6-e7597a1390d3-kube-api-access-mgb8t\") pod \"node-ca-zxnfn\" (UID: \"c0e82032-5917-427c-b7b6-e7597a1390d3\") " pod="openshift-image-registry/node-ca-zxnfn" Jul 23 00:23:30.154219 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.154197 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n492\" (UniqueName: \"kubernetes.io/projected/312fe3a7-e877-41e5-aedf-48c7c7ea2efa-kube-api-access-7n492\") pod \"aws-ebs-csi-driver-node-282sh\" (UID: \"312fe3a7-e877-41e5-aedf-48c7c7ea2efa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.154625 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.154580 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqsct\" (UniqueName: \"kubernetes.io/projected/581c6c94-dd79-49de-9cc9-fcbd6f40bcaf-kube-api-access-sqsct\") pod \"multus-tf94d\" (UID: \"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf\") " pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.154832 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.154800 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdgmw\" (UniqueName: \"kubernetes.io/projected/a62da51e-b639-41a0-b6dc-0d48da42b3c5-kube-api-access-mdgmw\") pod \"iptables-alerter-xqsl8\" (UID: \"a62da51e-b639-41a0-b6dc-0d48da42b3c5\") " pod="openshift-network-operator/iptables-alerter-xqsl8" Jul 23 00:23:30.156119 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.156076 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsjpc\" (UniqueName: \"kubernetes.io/projected/91698027-ce8c-4729-86e9-6eb62632951d-kube-api-access-qsjpc\") pod \"node-resolver-jl4hg\" (UID: \"91698027-ce8c-4729-86e9-6eb62632951d\") " pod="openshift-dns/node-resolver-jl4hg" Jul 23 00:23:30.157012 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.156885 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb7bl\" (UniqueName: \"kubernetes.io/projected/c3e1769c-38a9-4493-8e79-c758ea29b40a-kube-api-access-bb7bl\") pod \"multus-additional-cni-plugins-h6c6l\" (UID: \"c3e1769c-38a9-4493-8e79-c758ea29b40a\") " pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.157362 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.157330 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6vhf\" (UniqueName: \"kubernetes.io/projected/dc61300d-11be-4c5d-a1fd-6e305d4e193a-kube-api-access-r6vhf\") pod \"tuned-dxrmh\" (UID: \"dc61300d-11be-4c5d-a1fd-6e305d4e193a\") " pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.157952 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.157927 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr65w\" (UniqueName: \"kubernetes.io/projected/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-kube-api-access-lr65w\") pod \"network-metrics-daemon-mx6kd\" (UID: \"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246\") " pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:30.187562 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.187524 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:30.201170 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:30.201122 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4123100_2f7f_4b30_834f_04d2de11303b.slice/crio-6d44c56437bcb372fb995b7dd5bacfcc24099fd32452853c7f072e7b176dba3b WatchSource:0}: Error finding container 6d44c56437bcb372fb995b7dd5bacfcc24099fd32452853c7f072e7b176dba3b: Status 404 returned error can't find the container with id 6d44c56437bcb372fb995b7dd5bacfcc24099fd32452853c7f072e7b176dba3b Jul 23 00:23:30.230161 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.229959 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-82f2h" Jul 23 00:23:30.238984 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:30.238954 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6411344d_2ba4_418f_9b2c_e5cd631325b6.slice/crio-c3c6ec5aa1a53821aa1039bc2f5b790107c3833c7da095f0bc71a0f80852e78a WatchSource:0}: Error finding container c3c6ec5aa1a53821aa1039bc2f5b790107c3833c7da095f0bc71a0f80852e78a: Status 404 returned error can't find the container with id c3c6ec5aa1a53821aa1039bc2f5b790107c3833c7da095f0bc71a0f80852e78a Jul 23 00:23:30.261570 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.261526 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xqsl8" Jul 23 00:23:30.267819 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.267803 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zxnfn" Jul 23 00:23:30.290248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.290189 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" Jul 23 00:23:30.298004 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.297979 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" Jul 23 00:23:30.306451 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.306428 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jl4hg" Jul 23 00:23:30.327391 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.327362 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 00:23:30.340154 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.340133 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h6c6l" Jul 23 00:23:30.361463 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.361439 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tf94d" Jul 23 00:23:30.649483 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.649452 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs\") pod \"network-metrics-daemon-mx6kd\" (UID: \"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246\") " pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:30.649483 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.649486 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:30.649679 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.649615 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:23:30.649679 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.649661 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:23:30.649679 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.649679 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret podName:75dffbc4-21a5-4051-8cae-f7f9988f6630 nodeName:}" failed. No retries permitted until 2026-07-23 00:23:31.649663912 +0000 UTC m=+4.145332604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret") pod "global-pull-secret-syncer-njhfs" (UID: "75dffbc4-21a5-4051-8cae-f7f9988f6630") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:23:30.649794 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.649731 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs podName:9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246 nodeName:}" failed. No retries permitted until 2026-07-23 00:23:31.649711787 +0000 UTC m=+4.145380480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs") pod "network-metrics-daemon-mx6kd" (UID: "9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:23:30.750200 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.750152 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb86z\" (UniqueName: \"kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z\") pod \"network-check-target-vvs7d\" (UID: \"85661880-f744-4f0a-a606-822e19bba144\") " pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:30.750391 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.750368 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:23:30.750460 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.750404 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:23:30.750460 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.750420 2564 projected.go:196] Error preparing data for projected volume kube-api-access-hb86z for pod openshift-network-diagnostics/network-check-target-vvs7d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:23:30.750565 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:30.750493 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z podName:85661880-f744-4f0a-a606-822e19bba144 nodeName:}" failed. No retries permitted until 2026-07-23 00:23:31.750473336 +0000 UTC m=+4.246142030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb86z" (UniqueName: "kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z") pod "network-check-target-vvs7d" (UID: "85661880-f744-4f0a-a606-822e19bba144") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:23:30.928925 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.928801 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 00:18:30 +0000 UTC" deadline="2028-04-12 00:02:47.003930926 +0000 UTC" Jul 23 00:23:30.928925 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:30.928875 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15095h39m16.075060507s" Jul 23 00:23:31.067317 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:31.067277 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:31.067874 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:31.067450 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:31.076456 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:31.076382 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-82f2h" event={"ID":"6411344d-2ba4-418f-9b2c-e5cd631325b6","Type":"ContainerStarted","Data":"c3c6ec5aa1a53821aa1039bc2f5b790107c3833c7da095f0bc71a0f80852e78a"} Jul 23 00:23:31.078087 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:31.078057 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" event={"ID":"e4123100-2f7f-4b30-834f-04d2de11303b","Type":"ContainerStarted","Data":"6d44c56437bcb372fb995b7dd5bacfcc24099fd32452853c7f072e7b176dba3b"} Jul 23 00:23:31.461932 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:31.461796 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312fe3a7_e877_41e5_aedf_48c7c7ea2efa.slice/crio-6faaf1d853917b78ed543acd441363764c7b989d29365c0625a14c6cdd64fe23 WatchSource:0}: Error finding container 6faaf1d853917b78ed543acd441363764c7b989d29365c0625a14c6cdd64fe23: Status 404 returned error can't find the container with id 6faaf1d853917b78ed543acd441363764c7b989d29365c0625a14c6cdd64fe23 Jul 23 00:23:31.470675 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:31.470642 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e1769c_38a9_4493_8e79_c758ea29b40a.slice/crio-7317762a98d0b81f5acd49400402068d501cc1d28f7ff5191328097154fce087 WatchSource:0}: Error finding container 7317762a98d0b81f5acd49400402068d501cc1d28f7ff5191328097154fce087: Status 404 returned error can't find the container with id 7317762a98d0b81f5acd49400402068d501cc1d28f7ff5191328097154fce087 Jul 23 00:23:31.472247 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:31.472220 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581c6c94_dd79_49de_9cc9_fcbd6f40bcaf.slice/crio-14518584b795d383136cef1b4514bcd3ea48c5562eb2fb3048dbafabb0d70d4e WatchSource:0}: Error finding container 14518584b795d383136cef1b4514bcd3ea48c5562eb2fb3048dbafabb0d70d4e: Status 404 returned error can't find the container with id 14518584b795d383136cef1b4514bcd3ea48c5562eb2fb3048dbafabb0d70d4e Jul 23 00:23:31.476061 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:23:31.475918 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc61300d_11be_4c5d_a1fd_6e305d4e193a.slice/crio-c8190ab9d6a133756fc470873ed0fe8d7afff9a2b186d8d581b04ffc3cf8f580 WatchSource:0}: Error finding container c8190ab9d6a133756fc470873ed0fe8d7afff9a2b186d8d581b04ffc3cf8f580: Status 404 returned error can't find the container with id c8190ab9d6a133756fc470873ed0fe8d7afff9a2b186d8d581b04ffc3cf8f580 Jul 23 00:23:31.658439 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:31.658069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs\") pod \"network-metrics-daemon-mx6kd\" (UID: \"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246\") " pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:31.658439 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:31.658238 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:23:31.658687 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:31.658484 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs podName:9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246 nodeName:}" failed. No retries permitted until 2026-07-23 00:23:33.658452803 +0000 UTC m=+6.154121500 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs") pod "network-metrics-daemon-mx6kd" (UID: "9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:23:31.658687 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:31.658505 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:23:31.658687 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:31.658560 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret podName:75dffbc4-21a5-4051-8cae-f7f9988f6630 nodeName:}" failed. No retries permitted until 2026-07-23 00:23:33.658544558 +0000 UTC m=+6.154213274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret") pod "global-pull-secret-syncer-njhfs" (UID: "75dffbc4-21a5-4051-8cae-f7f9988f6630") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:23:31.658687 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:31.658390 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:31.760346 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:31.759702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb86z\" (UniqueName: \"kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z\") pod \"network-check-target-vvs7d\" (UID: \"85661880-f744-4f0a-a606-822e19bba144\") " pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:31.760346 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:31.759916 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:23:31.760346 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:31.759935 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:23:31.760346 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:31.759948 2564 projected.go:196] Error preparing data for projected volume kube-api-access-hb86z for pod openshift-network-diagnostics/network-check-target-vvs7d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:23:31.760346 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:31.760009 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z podName:85661880-f744-4f0a-a606-822e19bba144 nodeName:}" failed. No retries permitted until 2026-07-23 00:23:33.759990232 +0000 UTC m=+6.255658945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb86z" (UniqueName: "kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z") pod "network-check-target-vvs7d" (UID: "85661880-f744-4f0a-a606-822e19bba144") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:23:31.929576 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:31.929456 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 00:18:30 +0000 UTC" deadline="2028-02-04 15:40:34.451720005 +0000 UTC" Jul 23 00:23:31.929576 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:31.929499 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13479h17m2.522226908s" Jul 23 00:23:32.069491 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:32.069453 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:32.070142 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:32.069597 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:32.070142 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:32.069751 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:32.070142 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:32.069852 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:32.082619 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:32.082340 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-1.ec2.internal" event={"ID":"4359bec0e352431350db386bdadf6398","Type":"ContainerStarted","Data":"1a15a6987205d341aa3b03cc341197276eef334b2a731f68278fa5ce8691d822"} Jul 23 00:23:32.084287 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:32.084260 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" event={"ID":"dc61300d-11be-4c5d-a1fd-6e305d4e193a","Type":"ContainerStarted","Data":"c8190ab9d6a133756fc470873ed0fe8d7afff9a2b186d8d581b04ffc3cf8f580"} Jul 23 00:23:32.086432 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:32.086347 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" event={"ID":"312fe3a7-e877-41e5-aedf-48c7c7ea2efa","Type":"ContainerStarted","Data":"6faaf1d853917b78ed543acd441363764c7b989d29365c0625a14c6cdd64fe23"} Jul 23 00:23:32.088958 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:32.088539 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal" event={"ID":"a315493ff7ec57552d98ccbaa050d33e","Type":"ContainerStarted","Data":"5481eb7254947f4f8d0e7c61992e0c6eac23001b473f6a063e70617e040bd480"} Jul 23 00:23:32.091383 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:32.091324 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tf94d" event={"ID":"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf","Type":"ContainerStarted","Data":"14518584b795d383136cef1b4514bcd3ea48c5562eb2fb3048dbafabb0d70d4e"} Jul 23 00:23:32.092897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:32.092868 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jl4hg" event={"ID":"91698027-ce8c-4729-86e9-6eb62632951d","Type":"ContainerStarted","Data":"ade33c14ff5e2607821b254d788c1e49010958b1b11cafc2bcfeac1e8495e035"} Jul 23 00:23:32.094580 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:32.094524 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6c6l" event={"ID":"c3e1769c-38a9-4493-8e79-c758ea29b40a","Type":"ContainerStarted","Data":"7317762a98d0b81f5acd49400402068d501cc1d28f7ff5191328097154fce087"} Jul 23 00:23:32.097493 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:32.096555 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zxnfn" event={"ID":"c0e82032-5917-427c-b7b6-e7597a1390d3","Type":"ContainerStarted","Data":"87b9ce59db26d44c671958175a09d01dc7365e852030af57d354efd91e7f19c6"} Jul 23 00:23:32.097493 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:32.096882 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-1.ec2.internal" podStartSLOduration=3.096864931 podStartE2EDuration="3.096864931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:32.096573498 +0000 UTC m=+4.592242216" watchObservedRunningTime="2026-07-23 00:23:32.096864931 +0000 UTC m=+4.592533649" Jul 23 00:23:32.098284 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:32.098258 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xqsl8" event={"ID":"a62da51e-b639-41a0-b6dc-0d48da42b3c5","Type":"ContainerStarted","Data":"7518c182ad3ddd0c5e52c06ebf47a1e5d5dd2b64c13e7a82f67651eb198ae4b3"} Jul 23 00:23:33.067217 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:33.067183 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:33.067416 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:33.067346 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:33.108595 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:33.107710 2564 generic.go:356] "Generic (PLEG): container finished" podID="a315493ff7ec57552d98ccbaa050d33e" containerID="5481eb7254947f4f8d0e7c61992e0c6eac23001b473f6a063e70617e040bd480" exitCode=0 Jul 23 00:23:33.108595 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:33.108374 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal" event={"ID":"a315493ff7ec57552d98ccbaa050d33e","Type":"ContainerDied","Data":"5481eb7254947f4f8d0e7c61992e0c6eac23001b473f6a063e70617e040bd480"} Jul 23 00:23:33.114303 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:33.114192 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-82f2h" event={"ID":"6411344d-2ba4-418f-9b2c-e5cd631325b6","Type":"ContainerStarted","Data":"c62f8b09421289d34b51bd25ad42f01f47bfdbe9a691ce099b876d9a4ba3051d"} Jul 23 00:23:33.679800 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:33.678931 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs\") pod \"network-metrics-daemon-mx6kd\" (UID: \"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246\") " pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:33.679800 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:33.678981 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:33.679800 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:33.679123 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:23:33.679800 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:33.679185 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret podName:75dffbc4-21a5-4051-8cae-f7f9988f6630 nodeName:}" failed. No retries permitted until 2026-07-23 00:23:37.679167108 +0000 UTC m=+10.174835817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret") pod "global-pull-secret-syncer-njhfs" (UID: "75dffbc4-21a5-4051-8cae-f7f9988f6630") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:23:33.679800 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:33.679615 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:23:33.679800 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:33.679742 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs podName:9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246 nodeName:}" failed. No retries permitted until 2026-07-23 00:23:37.679718584 +0000 UTC m=+10.175387283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs") pod "network-metrics-daemon-mx6kd" (UID: "9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:23:33.780331 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:33.779723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb86z\" (UniqueName: \"kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z\") pod \"network-check-target-vvs7d\" (UID: \"85661880-f744-4f0a-a606-822e19bba144\") " pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:33.780331 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:33.779881 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:23:33.780331 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:33.779905 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:23:33.780331 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:33.779919 2564 projected.go:196] Error preparing data for projected volume kube-api-access-hb86z for pod openshift-network-diagnostics/network-check-target-vvs7d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:23:33.780331 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:33.779977 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z podName:85661880-f744-4f0a-a606-822e19bba144 nodeName:}" failed. No retries permitted until 2026-07-23 00:23:37.779959848 +0000 UTC m=+10.275628560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb86z" (UniqueName: "kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z") pod "network-check-target-vvs7d" (UID: "85661880-f744-4f0a-a606-822e19bba144") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:23:34.068142 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:34.067776 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:34.068142 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:34.067944 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:34.068142 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:34.067955 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:34.068142 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:34.068044 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:35.067147 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:35.067102 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:35.067576 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:35.067273 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:35.444468 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:35.444423 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-82f2h" Jul 23 00:23:35.445099 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:35.445070 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-82f2h" Jul 23 00:23:35.459760 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:35.459706 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-82f2h" podStartSLOduration=5.4084950880000005 podStartE2EDuration="7.459687407s" totalImagesPullingTime="2.051192319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:33.141284262 +0000 UTC m=+5.636952980" watchObservedRunningTime="2026-07-23 00:23:35.459687407 +0000 UTC m=+7.955356123" Jul 23 00:23:36.070599 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:36.070506 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:36.071079 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:36.070642 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:36.071079 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:36.071044 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:36.071202 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:36.071132 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:37.067871 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:37.067836 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:37.068056 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:37.067987 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:37.720366 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:37.720328 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs\") pod \"network-metrics-daemon-mx6kd\" (UID: \"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246\") " pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:37.720895 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:37.720378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:37.720895 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:37.720513 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:23:37.720895 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:37.720516 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:23:37.720895 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:37.720601 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret podName:75dffbc4-21a5-4051-8cae-f7f9988f6630 nodeName:}" failed. No retries permitted until 2026-07-23 00:23:45.720582043 +0000 UTC m=+18.216250750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret") pod "global-pull-secret-syncer-njhfs" (UID: "75dffbc4-21a5-4051-8cae-f7f9988f6630") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:23:37.720895 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:37.720622 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs podName:9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246 nodeName:}" failed. No retries permitted until 2026-07-23 00:23:45.720612305 +0000 UTC m=+18.216281014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs") pod "network-metrics-daemon-mx6kd" (UID: "9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:23:37.821651 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:37.821613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb86z\" (UniqueName: \"kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z\") pod \"network-check-target-vvs7d\" (UID: \"85661880-f744-4f0a-a606-822e19bba144\") " pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:37.821849 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:37.821781 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:23:37.821849 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:37.821806 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:23:37.821849 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:37.821819 2564 projected.go:196] Error preparing data for projected volume kube-api-access-hb86z for pod openshift-network-diagnostics/network-check-target-vvs7d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:23:37.822020 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:37.821888 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z podName:85661880-f744-4f0a-a606-822e19bba144 nodeName:}" failed. No retries permitted until 2026-07-23 00:23:45.821867261 +0000 UTC m=+18.317535978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb86z" (UniqueName: "kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z") pod "network-check-target-vvs7d" (UID: "85661880-f744-4f0a-a606-822e19bba144") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:23:38.070778 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:38.070691 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:38.070951 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:38.070820 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:38.071164 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:38.070698 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:38.071296 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:38.071270 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:39.067328 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:39.066809 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:39.067328 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:39.066960 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:40.066834 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:40.066801 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:40.067028 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:40.066953 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:40.067199 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:40.067179 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:40.067367 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:40.067289 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:41.067434 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:41.066937 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:41.067434 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:41.067079 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:42.067280 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:42.067253 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:42.067417 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:42.067254 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:42.067417 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:42.067361 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:42.067792 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:42.067427 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:43.066518 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:43.066488 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:43.066691 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:43.066631 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:44.067454 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:44.067374 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:44.067927 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:44.067506 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:44.067927 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:44.067564 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:44.067927 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:44.067694 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:45.066705 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:45.066674 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:45.066914 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:45.066807 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:45.785851 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:45.785813 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs\") pod \"network-metrics-daemon-mx6kd\" (UID: \"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246\") " pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:45.785851 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:45.785850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:45.786377 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:45.785969 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:23:45.786377 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:45.786029 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret podName:75dffbc4-21a5-4051-8cae-f7f9988f6630 nodeName:}" failed. No retries permitted until 2026-07-23 00:24:01.786011038 +0000 UTC m=+34.281679729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret") pod "global-pull-secret-syncer-njhfs" (UID: "75dffbc4-21a5-4051-8cae-f7f9988f6630") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:23:45.786377 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:45.785971 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:23:45.786377 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:45.786104 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs podName:9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246 nodeName:}" failed. No retries permitted until 2026-07-23 00:24:01.786090452 +0000 UTC m=+34.281759144 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs") pod "network-metrics-daemon-mx6kd" (UID: "9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:23:45.886828 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:45.886796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb86z\" (UniqueName: \"kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z\") pod \"network-check-target-vvs7d\" (UID: \"85661880-f744-4f0a-a606-822e19bba144\") " pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:45.887012 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:45.886958 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:23:45.887012 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:45.886979 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:23:45.887012 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:45.886990 2564 projected.go:196] Error preparing data for projected volume kube-api-access-hb86z for pod openshift-network-diagnostics/network-check-target-vvs7d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:23:45.887158 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:45.887053 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z podName:85661880-f744-4f0a-a606-822e19bba144 nodeName:}" failed. No retries permitted until 2026-07-23 00:24:01.887034838 +0000 UTC m=+34.382703552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb86z" (UniqueName: "kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z") pod "network-check-target-vvs7d" (UID: "85661880-f744-4f0a-a606-822e19bba144") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:23:46.066543 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:46.066451 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:46.066692 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:46.066598 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:46.066692 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:46.066604 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:46.066692 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:46.066683 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:47.067452 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:47.067177 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:47.067452 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:47.067348 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:48.067541 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:48.067506 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:48.068051 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:48.067621 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:48.068051 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:48.067675 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:48.068051 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:48.067753 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:49.066656 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:49.066625 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:49.066812 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:49.066753 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:50.066698 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:50.066673 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:50.067089 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:50.066782 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:50.067089 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:50.066851 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:50.067089 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:50.066947 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:51.067090 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.066729 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:51.067820 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:51.067197 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:51.152365 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.152334 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal" event={"ID":"a315493ff7ec57552d98ccbaa050d33e","Type":"ContainerStarted","Data":"4b3ea8723c7bca254e6c1326d8922fb912cd979fe79062ae970ca814de8fcb6c"} Jul 23 00:23:51.153975 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.153951 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tf94d" event={"ID":"581c6c94-dd79-49de-9cc9-fcbd6f40bcaf","Type":"ContainerStarted","Data":"ec2f8a8ef9c84ac2868b46192774027996f4e0abc336f0abc0a249778f96dfae"} Jul 23 00:23:51.155257 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.155221 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jl4hg" event={"ID":"91698027-ce8c-4729-86e9-6eb62632951d","Type":"ContainerStarted","Data":"4de07ff3650f4136344ddc88d338872229c068d0553eb1a9650e4e59dc3f3ef0"} Jul 23 00:23:51.156542 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.156509 2564 generic.go:356] "Generic (PLEG): container finished" podID="c3e1769c-38a9-4493-8e79-c758ea29b40a" containerID="83fc7d76459a1b5480d6e179f2a798d22523402004193790bca924a9f839730b" exitCode=0 Jul 23 00:23:51.156633 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.156552 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6c6l" event={"ID":"c3e1769c-38a9-4493-8e79-c758ea29b40a","Type":"ContainerDied","Data":"83fc7d76459a1b5480d6e179f2a798d22523402004193790bca924a9f839730b"} Jul 23 00:23:51.158087 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.158039 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zxnfn" event={"ID":"c0e82032-5917-427c-b7b6-e7597a1390d3","Type":"ContainerStarted","Data":"efa2ce5240022fb93cb88dfad53bdccef64dbce8d6d04f8157a6f2c22fc8fa52"} Jul 23 00:23:51.163169 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.163153 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:23:51.163461 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.163444 2564 generic.go:356] "Generic (PLEG): container finished" podID="e4123100-2f7f-4b30-834f-04d2de11303b" containerID="4238514882c75d80532f749218ffbcf94f4195a20581728fff7f9dff13706d5f" exitCode=1 Jul 23 00:23:51.163506 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.163487 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" event={"ID":"e4123100-2f7f-4b30-834f-04d2de11303b","Type":"ContainerStarted","Data":"445017d09783a323be482f53283159af5764eefb0d0ec12d0f6acb6d710bbbe1"} Jul 23 00:23:51.163579 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.163565 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" event={"ID":"e4123100-2f7f-4b30-834f-04d2de11303b","Type":"ContainerStarted","Data":"ec12dabc8255fe6a582161ff7de44b623c75bf7f84f8e33848d024e2d8886cf5"} Jul 23 00:23:51.163619 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.163609 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" event={"ID":"e4123100-2f7f-4b30-834f-04d2de11303b","Type":"ContainerStarted","Data":"0af43206de134ead773be6a5bc8785ac9ff9c0b3a0de68a095c13751399170e2"} Jul 23 00:23:51.163689 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.163675 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" event={"ID":"e4123100-2f7f-4b30-834f-04d2de11303b","Type":"ContainerStarted","Data":"d19eaefe69bc9015be119f75579bd4adce7d98e4f60ed9ed2b0eb82c758a694e"} Jul 23 00:23:51.163737 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.163727 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" event={"ID":"e4123100-2f7f-4b30-834f-04d2de11303b","Type":"ContainerDied","Data":"4238514882c75d80532f749218ffbcf94f4195a20581728fff7f9dff13706d5f"} Jul 23 00:23:51.163785 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.163773 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" event={"ID":"e4123100-2f7f-4b30-834f-04d2de11303b","Type":"ContainerStarted","Data":"6cdec928db4f0df1f0b512da5a594973e43429a70713b72335e19d9abc40b4da"} Jul 23 00:23:51.164902 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.164885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" event={"ID":"dc61300d-11be-4c5d-a1fd-6e305d4e193a","Type":"ContainerStarted","Data":"23beb28d416dffc204a3273b342982c85b8d3094c691da617a8a3093e918e97a"} Jul 23 00:23:51.166228 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.166210 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" event={"ID":"312fe3a7-e877-41e5-aedf-48c7c7ea2efa","Type":"ContainerStarted","Data":"568dcc7baf4582b76be9df532888098f145f469d58abd4994b73243857b01ca7"} Jul 23 00:23:51.166519 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.166478 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-1.ec2.internal" podStartSLOduration=22.166467521 podStartE2EDuration="22.166467521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:51.166462839 +0000 UTC m=+23.662131552" watchObservedRunningTime="2026-07-23 00:23:51.166467521 +0000 UTC m=+23.662136233" Jul 23 00:23:51.179807 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.179769 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zxnfn" podStartSLOduration=3.453720667 podStartE2EDuration="22.179758484s" totalImagesPullingTime="18.726037817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:51.179689518 +0000 UTC m=+23.675358229" watchObservedRunningTime="2026-07-23 00:23:51.179758484 +0000 UTC m=+23.675427196" Jul 23 00:23:51.195489 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.195448 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jl4hg" podStartSLOduration=3.475710405 podStartE2EDuration="22.195437407s" totalImagesPullingTime="18.719727002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:51.195289908 +0000 UTC m=+23.690958622" watchObservedRunningTime="2026-07-23 00:23:51.195437407 +0000 UTC m=+23.691106119" Jul 23 00:23:51.240110 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.240073 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tf94d" podStartSLOduration=3.47440882 podStartE2EDuration="22.240062779s" totalImagesPullingTime="18.765653959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:51.239656968 +0000 UTC m=+23.735325681" watchObservedRunningTime="2026-07-23 00:23:51.240062779 +0000 UTC m=+23.735731492" Jul 23 00:23:51.256917 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.256879 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dxrmh" podStartSLOduration=3.541601605 podStartE2EDuration="22.256868658s" totalImagesPullingTime="18.715267053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:51.256505548 +0000 UTC m=+23.752174261" watchObservedRunningTime="2026-07-23 00:23:51.256868658 +0000 UTC m=+23.752537371" Jul 23 00:23:51.362629 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.362605 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 00:23:51.957192 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.957066 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T00:23:51.362624625Z","UUID":"a3b7117f-4b6d-45cf-ac6f-2f4d83f506db","Handler":null,"Name":"","Endpoint":""} Jul 23 00:23:51.959792 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.959440 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 00:23:51.959792 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:51.959484 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 00:23:52.070606 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:52.070580 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:52.071020 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:52.070580 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:52.071020 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:52.070717 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:52.071020 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:52.070784 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:52.171447 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:52.171390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" event={"ID":"312fe3a7-e877-41e5-aedf-48c7c7ea2efa","Type":"ContainerStarted","Data":"4b18e417e99f0c2bc1c0ac96e39f79715474aeef7506f3803f0daed304020a74"} Jul 23 00:23:52.173205 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:52.173175 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xqsl8" event={"ID":"a62da51e-b639-41a0-b6dc-0d48da42b3c5","Type":"ContainerStarted","Data":"21d6079465775717a69bde6b3b6f8fea2776ef1fa8f23cc14006bf5567857600"} Jul 23 00:23:52.326072 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:52.325989 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:52.416644 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:52.416611 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:52.582699 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:52.582437 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:52.637109 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:52.637075 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:53.066837 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:53.066818 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:53.066965 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:53.066936 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:53.179221 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:53.179170 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:23:53.179720 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:53.179617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" event={"ID":"e4123100-2f7f-4b30-834f-04d2de11303b","Type":"ContainerStarted","Data":"5a348b7c2163fa101f6d58e8104dba86b799eab35dc11b81f8ec9711b0553b79"} Jul 23 00:23:53.181902 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:53.181873 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" event={"ID":"312fe3a7-e877-41e5-aedf-48c7c7ea2efa","Type":"ContainerStarted","Data":"98cbd21f60dbe9d7f2fb37c2fc2caa025498dc3ced38135aae97dbf5a0709e4a"} Jul 23 00:23:53.203835 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:53.203791 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-282sh" podStartSLOduration=3.494966567 podStartE2EDuration="24.203778293s" totalImagesPullingTime="20.708811726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:53.203498244 +0000 UTC m=+25.699166957" watchObservedRunningTime="2026-07-23 00:23:53.203778293 +0000 UTC m=+25.699447005" Jul 23 00:23:53.204105 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:53.204035 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xqsl8" podStartSLOduration=5.439671957 podStartE2EDuration="24.204024247s" totalImagesPullingTime="18.76435229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:52.187591162 +0000 UTC m=+24.683259876" watchObservedRunningTime="2026-07-23 00:23:53.204024247 +0000 UTC m=+25.699692960" Jul 23 00:23:54.066538 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:54.066493 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:54.066752 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:54.066634 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:54.066752 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:54.066690 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:54.066867 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:54.066784 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:55.066660 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:55.066626 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:55.067296 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:55.066763 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:55.208501 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:55.208461 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:55.259840 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:55.259777 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:55.432644 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:55.432258 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:55.490332 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:55.490286 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:56.067048 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:56.067025 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:56.067547 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:56.067119 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:56.067547 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:56.067155 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:56.067547 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:56.067203 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:56.192024 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:56.192005 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:23:56.192380 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:56.192356 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" event={"ID":"e4123100-2f7f-4b30-834f-04d2de11303b","Type":"ContainerStarted","Data":"75be71a93e9f21a7a131f65b6be783b1ed71c3d904dce332729ec3368df26b24"} Jul 23 00:23:56.192633 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:56.192600 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:56.192704 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:56.192686 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:56.192804 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:56.192788 2564 scope.go:117] "RemoveContainer" containerID="4238514882c75d80532f749218ffbcf94f4195a20581728fff7f9dff13706d5f" Jul 23 00:23:56.194320 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:56.194301 2564 generic.go:356] "Generic (PLEG): container finished" podID="c3e1769c-38a9-4493-8e79-c758ea29b40a" containerID="47a13c1d89de0ba7192a5605f188554406cc562520f91e330b611367e6f72a8b" exitCode=0 Jul 23 00:23:56.194408 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:56.194330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6c6l" event={"ID":"c3e1769c-38a9-4493-8e79-c758ea29b40a","Type":"ContainerDied","Data":"47a13c1d89de0ba7192a5605f188554406cc562520f91e330b611367e6f72a8b"} Jul 23 00:23:56.210461 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:56.210442 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:57.066988 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.066962 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:57.067192 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:57.067064 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:57.199997 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.199968 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:23:57.200349 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.200327 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" event={"ID":"e4123100-2f7f-4b30-834f-04d2de11303b","Type":"ContainerStarted","Data":"60f4ba9d4ec571dc897bd5af4bd56c82072687f493aeaafaa2baf8b82724e4db"} Jul 23 00:23:57.200676 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.200650 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:57.214673 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.214648 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:23:57.229107 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.229065 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" podStartSLOduration=9.022734858 podStartE2EDuration="29.229052637s" totalImagesPullingTime="20.206317779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:23:57.227188527 +0000 UTC m=+29.722857239" watchObservedRunningTime="2026-07-23 00:23:57.229052637 +0000 UTC m=+29.724721349" Jul 23 00:23:57.652147 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.652103 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-njhfs"] Jul 23 00:23:57.652338 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.652281 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:57.652432 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:57.652411 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:57.652621 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.652600 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vvs7d"] Jul 23 00:23:57.652698 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.652685 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:57.652774 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:57.652757 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:57.654104 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.654076 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mx6kd"] Jul 23 00:23:57.654210 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.654165 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:23:57.654268 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:57.654251 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:23:57.847096 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.847070 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-82f2h" Jul 23 00:23:57.847264 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.847201 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 23 00:23:57.847612 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.847590 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-82f2h" Jul 23 00:23:57.906194 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:57.906134 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:23:58.204735 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:58.204671 2564 generic.go:356] "Generic (PLEG): container finished" podID="c3e1769c-38a9-4493-8e79-c758ea29b40a" containerID="11da94d5c49da52d9595b5e08cf02b35234e3d6ad175c195b5bacf673f8d9096" exitCode=0 Jul 23 00:23:58.205290 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:58.204755 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6c6l" event={"ID":"c3e1769c-38a9-4493-8e79-c758ea29b40a","Type":"ContainerDied","Data":"11da94d5c49da52d9595b5e08cf02b35234e3d6ad175c195b5bacf673f8d9096"} Jul 23 00:23:59.067249 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:59.067222 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:23:59.067360 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:59.067270 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:23:59.067416 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:59.067350 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:23:59.067491 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:23:59.067466 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:23:59.209811 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:59.209780 2564 generic.go:356] "Generic (PLEG): container finished" podID="c3e1769c-38a9-4493-8e79-c758ea29b40a" containerID="9b58896da62233991c27212de67e8d72900155283328ed7cd5c5c2b8b8541001" exitCode=0 Jul 23 00:23:59.210162 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:23:59.209861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6c6l" event={"ID":"c3e1769c-38a9-4493-8e79-c758ea29b40a","Type":"ContainerDied","Data":"9b58896da62233991c27212de67e8d72900155283328ed7cd5c5c2b8b8541001"} Jul 23 00:24:00.067362 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:00.067327 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:24:00.067546 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:00.067468 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:24:01.066783 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:01.066753 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:24:01.067429 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:01.066753 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:24:01.067429 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:01.066870 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vvs7d" podUID="85661880-f744-4f0a-a606-822e19bba144" Jul 23 00:24:01.067429 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:01.066986 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-njhfs" podUID="75dffbc4-21a5-4051-8cae-f7f9988f6630" Jul 23 00:24:01.798953 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:01.798869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs\") pod \"network-metrics-daemon-mx6kd\" (UID: \"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246\") " pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:24:01.798953 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:01.798902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:24:01.799129 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:01.799020 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 00:24:01.799129 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:01.799028 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:24:01.799129 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:01.799100 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret podName:75dffbc4-21a5-4051-8cae-f7f9988f6630 nodeName:}" failed. No retries permitted until 2026-07-23 00:24:33.799080675 +0000 UTC m=+66.294749371 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret") pod "global-pull-secret-syncer-njhfs" (UID: "75dffbc4-21a5-4051-8cae-f7f9988f6630") : object "kube-system"/"original-pull-secret" not registered Jul 23 00:24:01.799129 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:01.799118 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs podName:9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246 nodeName:}" failed. No retries permitted until 2026-07-23 00:24:33.79910888 +0000 UTC m=+66.294777571 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs") pod "network-metrics-daemon-mx6kd" (UID: "9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 00:24:01.899861 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:01.899820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb86z\" (UniqueName: \"kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z\") pod \"network-check-target-vvs7d\" (UID: \"85661880-f744-4f0a-a606-822e19bba144\") " pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:24:01.900018 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:01.899957 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 00:24:01.900018 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:01.899977 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 00:24:01.900018 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:01.899989 2564 projected.go:196] Error preparing data for projected volume kube-api-access-hb86z for pod openshift-network-diagnostics/network-check-target-vvs7d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:24:01.900185 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:01.900049 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z podName:85661880-f744-4f0a-a606-822e19bba144 nodeName:}" failed. No retries permitted until 2026-07-23 00:24:33.900031599 +0000 UTC m=+66.395700307 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hb86z" (UniqueName: "kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z") pod "network-check-target-vvs7d" (UID: "85661880-f744-4f0a-a606-822e19bba144") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 00:24:02.067647 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.067561 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:24:02.068065 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:02.067701 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:24:02.849367 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.849176 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-1.ec2.internal" event="NodeReady" Jul 23 00:24:02.849573 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.849495 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 00:24:02.898106 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.898076 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2gcnv"] Jul 23 00:24:02.920167 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.919925 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kdnq4"] Jul 23 00:24:02.920167 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.920031 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:02.927659 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.927631 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 00:24:02.927826 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.927718 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhdn8\"" Jul 23 00:24:02.927826 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.927780 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 00:24:02.933750 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.933732 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2gcnv"] Jul 23 00:24:02.933863 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.933771 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kdnq4"] Jul 23 00:24:02.933863 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.933844 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kdnq4" Jul 23 00:24:02.937831 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.937775 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 00:24:02.937962 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.937943 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 00:24:02.938042 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.938028 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vdgqd\"" Jul 23 00:24:02.938245 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:02.938224 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 00:24:03.009561 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.009520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:03.009731 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.009574 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6qrs\" (UniqueName: \"kubernetes.io/projected/e5461539-d692-499b-9b64-4d5b6def6be5-kube-api-access-r6qrs\") pod \"ingress-canary-kdnq4\" (UID: \"e5461539-d692-499b-9b64-4d5b6def6be5\") " pod="openshift-ingress-canary/ingress-canary-kdnq4" Jul 23 00:24:03.009731 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.009594 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbtnz\" (UniqueName: \"kubernetes.io/projected/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-kube-api-access-sbtnz\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:03.009731 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.009616 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-config-volume\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:03.009731 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.009631 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-tmp-dir\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:03.009731 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.009657 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5461539-d692-499b-9b64-4d5b6def6be5-cert\") pod \"ingress-canary-kdnq4\" (UID: \"e5461539-d692-499b-9b64-4d5b6def6be5\") " pod="openshift-ingress-canary/ingress-canary-kdnq4" Jul 23 00:24:03.066622 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.066595 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:24:03.066780 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.066596 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:24:03.069624 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.069605 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 00:24:03.070005 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.069895 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c75lg\"" Jul 23 00:24:03.070005 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.069943 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 00:24:03.070005 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.069953 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 00:24:03.110920 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.110898 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:03.111029 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.110942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6qrs\" (UniqueName: \"kubernetes.io/projected/e5461539-d692-499b-9b64-4d5b6def6be5-kube-api-access-r6qrs\") pod \"ingress-canary-kdnq4\" (UID: \"e5461539-d692-499b-9b64-4d5b6def6be5\") " pod="openshift-ingress-canary/ingress-canary-kdnq4" Jul 23 00:24:03.111029 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.110987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbtnz\" (UniqueName: \"kubernetes.io/projected/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-kube-api-access-sbtnz\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:03.111029 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.111024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-config-volume\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:03.111175 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.111047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-tmp-dir\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:03.111175 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:03.111058 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:24:03.111175 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.111071 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5461539-d692-499b-9b64-4d5b6def6be5-cert\") pod \"ingress-canary-kdnq4\" (UID: \"e5461539-d692-499b-9b64-4d5b6def6be5\") " pod="openshift-ingress-canary/ingress-canary-kdnq4" Jul 23 00:24:03.111175 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:03.111118 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls podName:cba4baf3-9d99-4bb1-a822-8d80d49c1f9a nodeName:}" failed. No retries permitted until 2026-07-23 00:24:03.61109623 +0000 UTC m=+36.106764924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls") pod "dns-default-2gcnv" (UID: "cba4baf3-9d99-4bb1-a822-8d80d49c1f9a") : secret "dns-default-metrics-tls" not found Jul 23 00:24:03.111548 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.111515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-tmp-dir\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:03.111776 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.111740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-config-volume\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:03.115654 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.115637 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5461539-d692-499b-9b64-4d5b6def6be5-cert\") pod \"ingress-canary-kdnq4\" (UID: \"e5461539-d692-499b-9b64-4d5b6def6be5\") " pod="openshift-ingress-canary/ingress-canary-kdnq4" Jul 23 00:24:03.122201 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.122173 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbtnz\" (UniqueName: \"kubernetes.io/projected/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-kube-api-access-sbtnz\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:03.122297 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.122223 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6qrs\" (UniqueName: \"kubernetes.io/projected/e5461539-d692-499b-9b64-4d5b6def6be5-kube-api-access-r6qrs\") pod \"ingress-canary-kdnq4\" (UID: \"e5461539-d692-499b-9b64-4d5b6def6be5\") " pod="openshift-ingress-canary/ingress-canary-kdnq4" Jul 23 00:24:03.244239 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.244167 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kdnq4" Jul 23 00:24:03.615344 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:03.615271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:03.615470 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:03.615395 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:24:03.615470 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:03.615451 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls podName:cba4baf3-9d99-4bb1-a822-8d80d49c1f9a nodeName:}" failed. No retries permitted until 2026-07-23 00:24:04.615438266 +0000 UTC m=+37.111106957 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls") pod "dns-default-2gcnv" (UID: "cba4baf3-9d99-4bb1-a822-8d80d49c1f9a") : secret "dns-default-metrics-tls" not found Jul 23 00:24:04.066620 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:04.066582 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:24:04.069641 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:04.069614 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 00:24:04.069997 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:04.069697 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8gvmb\"" Jul 23 00:24:04.280927 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:04.280892 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kdnq4"] Jul 23 00:24:04.285522 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:24:04.285498 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5461539_d692_499b_9b64_4d5b6def6be5.slice/crio-bfd8d808365d16d09d20f975fde9ecbd3b82d61ff1dae244626834414488a48b WatchSource:0}: Error finding container bfd8d808365d16d09d20f975fde9ecbd3b82d61ff1dae244626834414488a48b: Status 404 returned error can't find the container with id bfd8d808365d16d09d20f975fde9ecbd3b82d61ff1dae244626834414488a48b Jul 23 00:24:04.400199 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:04.400176 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:04.451139 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:04.451113 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:04.624665 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:04.624492 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:04.624759 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:04.624623 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:24:04.624759 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:04.624742 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls podName:cba4baf3-9d99-4bb1-a822-8d80d49c1f9a nodeName:}" failed. No retries permitted until 2026-07-23 00:24:06.624726138 +0000 UTC m=+39.120394842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls") pod "dns-default-2gcnv" (UID: "cba4baf3-9d99-4bb1-a822-8d80d49c1f9a") : secret "dns-default-metrics-tls" not found Jul 23 00:24:05.228081 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:05.228050 2564 generic.go:356] "Generic (PLEG): container finished" podID="c3e1769c-38a9-4493-8e79-c758ea29b40a" containerID="51f0f6aa1d5573dedbe206c574ed628c6dae6a27264d1223d71f9950103352e8" exitCode=0 Jul 23 00:24:05.228508 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:05.228141 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6c6l" event={"ID":"c3e1769c-38a9-4493-8e79-c758ea29b40a","Type":"ContainerDied","Data":"51f0f6aa1d5573dedbe206c574ed628c6dae6a27264d1223d71f9950103352e8"} Jul 23 00:24:05.229643 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:05.229617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kdnq4" event={"ID":"e5461539-d692-499b-9b64-4d5b6def6be5","Type":"ContainerStarted","Data":"bfd8d808365d16d09d20f975fde9ecbd3b82d61ff1dae244626834414488a48b"} Jul 23 00:24:06.235578 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:06.235542 2564 generic.go:356] "Generic (PLEG): container finished" podID="c3e1769c-38a9-4493-8e79-c758ea29b40a" containerID="f23446bca4629211a4c9e4e959ead17f02515b4687a8bb8346c0836026eb4ff7" exitCode=0 Jul 23 00:24:06.236054 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:06.235612 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6c6l" event={"ID":"c3e1769c-38a9-4493-8e79-c758ea29b40a","Type":"ContainerDied","Data":"f23446bca4629211a4c9e4e959ead17f02515b4687a8bb8346c0836026eb4ff7"} Jul 23 00:24:06.361497 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:06.361449 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:06.417621 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:06.417585 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:06.642747 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:06.642727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:06.642855 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:06.642841 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:24:06.642915 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:06.642896 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls podName:cba4baf3-9d99-4bb1-a822-8d80d49c1f9a nodeName:}" failed. No retries permitted until 2026-07-23 00:24:10.642876325 +0000 UTC m=+43.138545016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls") pod "dns-default-2gcnv" (UID: "cba4baf3-9d99-4bb1-a822-8d80d49c1f9a") : secret "dns-default-metrics-tls" not found Jul 23 00:24:07.093653 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:07.093621 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:07.155748 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:07.155711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:07.241255 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:07.241186 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h6c6l" event={"ID":"c3e1769c-38a9-4493-8e79-c758ea29b40a","Type":"ContainerStarted","Data":"a685ecc5213c7226420e73de72b066dd927c9c3edb01ce42ef98bb56d2c3a253"} Jul 23 00:24:07.242716 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:07.242682 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kdnq4" event={"ID":"e5461539-d692-499b-9b64-4d5b6def6be5","Type":"ContainerStarted","Data":"18e4a38e759cbe655cabe7a5a8791a299294601dacbadcfa733a9d8f48610707"} Jul 23 00:24:07.265196 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:07.265156 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h6c6l" podStartSLOduration=8.280657883 podStartE2EDuration="38.265142863s" totalImagesPullingTime="29.98448498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:07.263584131 +0000 UTC m=+39.759252843" watchObservedRunningTime="2026-07-23 00:24:07.265142863 +0000 UTC m=+39.760811606" Jul 23 00:24:07.280757 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:07.280714 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kdnq4" podStartSLOduration=2.7973777699999998 podStartE2EDuration="5.280700791s" totalImagesPullingTime="2.483323021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:07.279906865 +0000 UTC m=+39.775575568" watchObservedRunningTime="2026-07-23 00:24:07.280700791 +0000 UTC m=+39.776369503" Jul 23 00:24:07.409158 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:07.409123 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:07.509891 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:07.509800 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:08.212196 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:08.212152 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:10.672964 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:10.672921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:10.673381 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:10.673039 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:24:10.673381 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:10.673093 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls podName:cba4baf3-9d99-4bb1-a822-8d80d49c1f9a nodeName:}" failed. No retries permitted until 2026-07-23 00:24:18.673079118 +0000 UTC m=+51.168747809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls") pod "dns-default-2gcnv" (UID: "cba4baf3-9d99-4bb1-a822-8d80d49c1f9a") : secret "dns-default-metrics-tls" not found Jul 23 00:24:17.317708 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:17.317668 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:17.418597 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:17.418549 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:18.730788 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:18.730742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:18.731192 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:18.730874 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:24:18.731192 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:18.730936 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls podName:cba4baf3-9d99-4bb1-a822-8d80d49c1f9a nodeName:}" failed. No retries permitted until 2026-07-23 00:24:34.730919815 +0000 UTC m=+67.226588507 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls") pod "dns-default-2gcnv" (UID: "cba4baf3-9d99-4bb1-a822-8d80d49c1f9a") : secret "dns-default-metrics-tls" not found Jul 23 00:24:28.453543 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:28.453488 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:28.520468 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:28.520441 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:29.220364 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:29.220341 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hq5hv" Jul 23 00:24:29.387262 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:29.387230 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:29.436626 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:29.436588 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:32.115490 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:32.115450 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:32.218429 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:32.218390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:33.833264 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:33.833230 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs\") pod \"network-metrics-daemon-mx6kd\" (UID: \"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246\") " pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:24:33.833264 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:33.833262 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:24:33.836257 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:33.836240 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 00:24:33.836307 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:33.836240 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 00:24:33.844371 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:33.844349 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 23 00:24:33.844434 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:33.844422 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs podName:9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246 nodeName:}" failed. No retries permitted until 2026-07-23 00:25:37.844406506 +0000 UTC m=+130.340075200 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs") pod "network-metrics-daemon-mx6kd" (UID: "9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246") : secret "metrics-daemon-secret" not found Jul 23 00:24:33.847461 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:33.847438 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/75dffbc4-21a5-4051-8cae-f7f9988f6630-original-pull-secret\") pod \"global-pull-secret-syncer-njhfs\" (UID: \"75dffbc4-21a5-4051-8cae-f7f9988f6630\") " pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:24:33.934421 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:33.934397 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb86z\" (UniqueName: \"kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z\") pod \"network-check-target-vvs7d\" (UID: \"85661880-f744-4f0a-a606-822e19bba144\") " pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:24:33.937169 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:33.937153 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 00:24:33.947463 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:33.947447 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 00:24:33.957748 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:33.957724 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb86z\" (UniqueName: \"kubernetes.io/projected/85661880-f744-4f0a-a606-822e19bba144-kube-api-access-hb86z\") pod \"network-check-target-vvs7d\" (UID: \"85661880-f744-4f0a-a606-822e19bba144\") " pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:24:33.981626 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:33.981602 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c75lg\"" Jul 23 00:24:33.984823 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:33.984808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-njhfs" Jul 23 00:24:33.989423 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:33.989410 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:24:34.112886 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:34.112857 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-njhfs"] Jul 23 00:24:34.116069 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:24:34.116045 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75dffbc4_21a5_4051_8cae_f7f9988f6630.slice/crio-af42fb9dd1a2774c9cbc7975209fbd225484011884ef3538fdac2439f3345db8 WatchSource:0}: Error finding container af42fb9dd1a2774c9cbc7975209fbd225484011884ef3538fdac2439f3345db8: Status 404 returned error can't find the container with id af42fb9dd1a2774c9cbc7975209fbd225484011884ef3538fdac2439f3345db8 Jul 23 00:24:34.124744 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:34.124720 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vvs7d"] Jul 23 00:24:34.127132 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:24:34.127113 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85661880_f744_4f0a_a606_822e19bba144.slice/crio-5f85f197ebcd2c0c5f7cc3316194a3fa2ebe803a5f055db485efbf4f98272f03 WatchSource:0}: Error finding container 5f85f197ebcd2c0c5f7cc3316194a3fa2ebe803a5f055db485efbf4f98272f03: Status 404 returned error can't find the container with id 5f85f197ebcd2c0c5f7cc3316194a3fa2ebe803a5f055db485efbf4f98272f03 Jul 23 00:24:34.307134 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:34.307102 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vvs7d" event={"ID":"85661880-f744-4f0a-a606-822e19bba144","Type":"ContainerStarted","Data":"5f85f197ebcd2c0c5f7cc3316194a3fa2ebe803a5f055db485efbf4f98272f03"} Jul 23 00:24:34.308155 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:34.308127 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-njhfs" event={"ID":"75dffbc4-21a5-4051-8cae-f7f9988f6630","Type":"ContainerStarted","Data":"af42fb9dd1a2774c9cbc7975209fbd225484011884ef3538fdac2439f3345db8"} Jul 23 00:24:34.738456 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:34.738414 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:24:34.738652 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:34.738572 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:24:34.738719 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:24:34.738678 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls podName:cba4baf3-9d99-4bb1-a822-8d80d49c1f9a nodeName:}" failed. No retries permitted until 2026-07-23 00:25:06.738658873 +0000 UTC m=+99.234327570 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls") pod "dns-default-2gcnv" (UID: "cba4baf3-9d99-4bb1-a822-8d80d49c1f9a") : secret "dns-default-metrics-tls" not found Jul 23 00:24:39.322574 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:39.322517 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-njhfs" event={"ID":"75dffbc4-21a5-4051-8cae-f7f9988f6630","Type":"ContainerStarted","Data":"77f870914014304d6c1d5b519715ba4621e4273dbaca8373ac362658c022536e"} Jul 23 00:24:39.324017 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:39.323994 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vvs7d" event={"ID":"85661880-f744-4f0a-a606-822e19bba144","Type":"ContainerStarted","Data":"65b60b5bb2e5018e9ef1188ebe7f790ed6de4f94592811c819bbbce032583073"} Jul 23 00:24:39.324131 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:39.324112 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:24:39.339459 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:39.339417 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-njhfs" podStartSLOduration=65.547187951 podStartE2EDuration="1m10.339405591s" totalImagesPullingTime="4.79221764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:39.337907463 +0000 UTC m=+71.833576177" watchObservedRunningTime="2026-07-23 00:24:39.339405591 +0000 UTC m=+71.835074340" Jul 23 00:24:39.352486 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:39.352447 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vvs7d" podStartSLOduration=65.582240323 podStartE2EDuration="1m10.352436676s" totalImagesPullingTime="4.770196353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:24:39.351941255 +0000 UTC m=+71.847609978" watchObservedRunningTime="2026-07-23 00:24:39.352436676 +0000 UTC m=+71.848105388" Jul 23 00:24:54.669942 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:54.669813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:24:54.725607 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:24:54.725569 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:25:06.739403 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:25:06.739367 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:25:06.739812 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:25:06.739481 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:25:06.739812 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:25:06.739564 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls podName:cba4baf3-9d99-4bb1-a822-8d80d49c1f9a nodeName:}" failed. No retries permitted until 2026-07-23 00:26:10.739545371 +0000 UTC m=+163.235214064 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls") pod "dns-default-2gcnv" (UID: "cba4baf3-9d99-4bb1-a822-8d80d49c1f9a") : secret "dns-default-metrics-tls" not found Jul 23 00:25:10.328972 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:25:10.328944 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vvs7d" Jul 23 00:25:37.844893 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:25:37.844850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs\") pod \"network-metrics-daemon-mx6kd\" (UID: \"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246\") " pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:25:37.845360 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:25:37.844998 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 23 00:25:37.845360 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:25:37.845065 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs podName:9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:39.845046059 +0000 UTC m=+252.340714753 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs") pod "network-metrics-daemon-mx6kd" (UID: "9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246") : secret "metrics-daemon-secret" not found Jul 23 00:25:52.629659 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:25:52.629626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:26:05.931574 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:26:05.931512 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2gcnv" podUID="cba4baf3-9d99-4bb1-a822-8d80d49c1f9a" Jul 23 00:26:06.527961 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:26:06.527933 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2gcnv" Jul 23 00:26:07.075217 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:26:07.075163 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mx6kd" podUID="9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246" Jul 23 00:26:10.754950 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:26:10.754913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:26:10.755343 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:26:10.755031 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 00:26:10.755343 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:26:10.755106 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls podName:cba4baf3-9d99-4bb1-a822-8d80d49c1f9a nodeName:}" failed. No retries permitted until 2026-07-23 00:28:12.755093248 +0000 UTC m=+285.250761939 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls") pod "dns-default-2gcnv" (UID: "cba4baf3-9d99-4bb1-a822-8d80d49c1f9a") : secret "dns-default-metrics-tls" not found Jul 23 00:26:21.067304 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:26:21.067217 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:26:57.036423 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:26:57.036381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:21.651064 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.651028 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77bf5fc84-r6bwx"] Jul 23 00:27:21.652667 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.652653 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.656065 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.656041 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 00:27:21.656195 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.656045 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 00:27:21.656195 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.656136 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6v7ht\"" Jul 23 00:27:21.656425 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.656406 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 00:27:21.662124 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.662104 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 00:27:21.679931 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.679909 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77bf5fc84-r6bwx"] Jul 23 00:27:21.684562 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.684522 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-bound-sa-token\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.684657 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.684578 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42166b50-1d69-4b6f-a926-0a9f61f76c60-ca-trust-extracted\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.684657 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.684596 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-certificates\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.684733 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.684680 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp95n\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-kube-api-access-gp95n\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.684733 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.684711 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42166b50-1d69-4b6f-a926-0a9f61f76c60-image-registry-private-configuration\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.684807 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.684747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.684807 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.684777 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42166b50-1d69-4b6f-a926-0a9f61f76c60-trusted-ca\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.684807 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.684792 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42166b50-1d69-4b6f-a926-0a9f61f76c60-installation-pull-secrets\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.740774 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.740750 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-h8gb6"] Jul 23 00:27:21.745265 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.745241 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-h8gb6" Jul 23 00:27:21.747135 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.747109 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6"] Jul 23 00:27:21.748120 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.748099 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hpt2n\"" Jul 23 00:27:21.748359 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.748344 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 23 00:27:21.748792 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.748771 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm"] Jul 23 00:27:21.748907 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.748888 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" Jul 23 00:27:21.750456 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.750437 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-wjhdx"] Jul 23 00:27:21.750572 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.750460 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:27:21.750641 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.750568 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" Jul 23 00:27:21.752132 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.752112 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.752304 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.752286 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 23 00:27:21.752437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.752417 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ql2b9\"" Jul 23 00:27:21.752906 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.752886 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 23 00:27:21.753200 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.753178 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:27:21.753554 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.753541 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 23 00:27:21.754634 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.754618 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 00:27:21.754931 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.754918 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 23 00:27:21.755085 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.755073 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-sxbdh\"" Jul 23 00:27:21.755130 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.755115 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tj2kx\"" Jul 23 00:27:21.755889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.755869 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 23 00:27:21.755993 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.755899 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 23 00:27:21.755993 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.755908 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 23 00:27:21.755993 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.755908 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:27:21.756113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.755902 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 00:27:21.762796 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.762776 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 23 00:27:21.764173 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.764154 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-h8gb6"] Jul 23 00:27:21.771522 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.771085 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6"] Jul 23 00:27:21.771972 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.771956 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm"] Jul 23 00:27:21.785877 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.785851 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/436060b9-c207-4a56-a70e-65e1a51c3b17-serving-cert\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.785976 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.785880 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf5354e7-1133-4900-93f6-dbf7fddeed5b-serving-cert\") pod \"service-ca-operator-5bcbc94c78-x8wz6\" (UID: \"bf5354e7-1133-4900-93f6-dbf7fddeed5b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" Jul 23 00:27:21.785976 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.785912 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42166b50-1d69-4b6f-a926-0a9f61f76c60-image-registry-private-configuration\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.785976 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.785942 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2k2l\" (UniqueName: \"kubernetes.io/projected/5068b739-0dd4-4654-a415-417a309565be-kube-api-access-w2k2l\") pod \"volume-data-source-validator-78cf4c6f6-h8gb6\" (UID: \"5068b739-0dd4-4654-a415-417a309565be\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-h8gb6" Jul 23 00:27:21.785976 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.785966 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b4zzm\" (UID: \"c0835e02-d417-48ce-8bc4-0e28b009faf7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" Jul 23 00:27:21.786178 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786020 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.786178 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786045 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/436060b9-c207-4a56-a70e-65e1a51c3b17-service-ca-bundle\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.786178 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42166b50-1d69-4b6f-a926-0a9f61f76c60-trusted-ca\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.786178 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786104 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42166b50-1d69-4b6f-a926-0a9f61f76c60-installation-pull-secrets\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.786178 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786130 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/436060b9-c207-4a56-a70e-65e1a51c3b17-snapshots\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.786178 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:21.786148 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:27:21.786178 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:21.786173 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77bf5fc84-r6bwx: secret "image-registry-tls" not found Jul 23 00:27:21.786178 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786153 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d69lm\" (UniqueName: \"kubernetes.io/projected/436060b9-c207-4a56-a70e-65e1a51c3b17-kube-api-access-d69lm\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.786561 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:21.786251 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls podName:42166b50-1d69-4b6f-a926-0a9f61f76c60 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:22.286231067 +0000 UTC m=+234.781899770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls") pod "image-registry-77bf5fc84-r6bwx" (UID: "42166b50-1d69-4b6f-a926-0a9f61f76c60") : secret "image-registry-tls" not found Jul 23 00:27:21.786561 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786274 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvt9k\" (UniqueName: \"kubernetes.io/projected/bf5354e7-1133-4900-93f6-dbf7fddeed5b-kube-api-access-rvt9k\") pod \"service-ca-operator-5bcbc94c78-x8wz6\" (UID: \"bf5354e7-1133-4900-93f6-dbf7fddeed5b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" Jul 23 00:27:21.786561 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-bound-sa-token\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.786561 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786392 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42166b50-1d69-4b6f-a926-0a9f61f76c60-ca-trust-extracted\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.786561 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-certificates\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.786561 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/436060b9-c207-4a56-a70e-65e1a51c3b17-tmp\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.786561 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf5354e7-1133-4900-93f6-dbf7fddeed5b-config\") pod \"service-ca-operator-5bcbc94c78-x8wz6\" (UID: \"bf5354e7-1133-4900-93f6-dbf7fddeed5b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" Jul 23 00:27:21.786900 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786569 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzhxw\" (UniqueName: \"kubernetes.io/projected/c0835e02-d417-48ce-8bc4-0e28b009faf7-kube-api-access-mzhxw\") pod \"cluster-samples-operator-79b7c55f87-b4zzm\" (UID: \"c0835e02-d417-48ce-8bc4-0e28b009faf7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" Jul 23 00:27:21.786900 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786625 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp95n\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-kube-api-access-gp95n\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.786900 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786657 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/436060b9-c207-4a56-a70e-65e1a51c3b17-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.786900 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786823 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42166b50-1d69-4b6f-a926-0a9f61f76c60-ca-trust-extracted\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.787056 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.786993 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-certificates\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.787098 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.787054 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42166b50-1d69-4b6f-a926-0a9f61f76c60-trusted-ca\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.788314 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.788297 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42166b50-1d69-4b6f-a926-0a9f61f76c60-installation-pull-secrets\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.788366 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.788343 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42166b50-1d69-4b6f-a926-0a9f61f76c60-image-registry-private-configuration\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.793172 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.793150 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-wjhdx"] Jul 23 00:27:21.807071 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.807049 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp95n\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-kube-api-access-gp95n\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.808332 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.808313 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-bound-sa-token\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:21.887042 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.887021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf5354e7-1133-4900-93f6-dbf7fddeed5b-config\") pod \"service-ca-operator-5bcbc94c78-x8wz6\" (UID: \"bf5354e7-1133-4900-93f6-dbf7fddeed5b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" Jul 23 00:27:21.887158 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.887061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzhxw\" (UniqueName: \"kubernetes.io/projected/c0835e02-d417-48ce-8bc4-0e28b009faf7-kube-api-access-mzhxw\") pod \"cluster-samples-operator-79b7c55f87-b4zzm\" (UID: \"c0835e02-d417-48ce-8bc4-0e28b009faf7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" Jul 23 00:27:21.887222 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.887175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/436060b9-c207-4a56-a70e-65e1a51c3b17-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.887222 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.887205 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/436060b9-c207-4a56-a70e-65e1a51c3b17-serving-cert\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.887321 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.887225 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf5354e7-1133-4900-93f6-dbf7fddeed5b-serving-cert\") pod \"service-ca-operator-5bcbc94c78-x8wz6\" (UID: \"bf5354e7-1133-4900-93f6-dbf7fddeed5b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" Jul 23 00:27:21.887321 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.887256 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2k2l\" (UniqueName: \"kubernetes.io/projected/5068b739-0dd4-4654-a415-417a309565be-kube-api-access-w2k2l\") pod \"volume-data-source-validator-78cf4c6f6-h8gb6\" (UID: \"5068b739-0dd4-4654-a415-417a309565be\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-h8gb6" Jul 23 00:27:21.887321 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.887281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b4zzm\" (UID: \"c0835e02-d417-48ce-8bc4-0e28b009faf7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" Jul 23 00:27:21.887461 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.887354 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/436060b9-c207-4a56-a70e-65e1a51c3b17-service-ca-bundle\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.887461 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.887397 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/436060b9-c207-4a56-a70e-65e1a51c3b17-snapshots\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.887461 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.887423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d69lm\" (UniqueName: \"kubernetes.io/projected/436060b9-c207-4a56-a70e-65e1a51c3b17-kube-api-access-d69lm\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.887461 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.887450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvt9k\" (UniqueName: \"kubernetes.io/projected/bf5354e7-1133-4900-93f6-dbf7fddeed5b-kube-api-access-rvt9k\") pod \"service-ca-operator-5bcbc94c78-x8wz6\" (UID: \"bf5354e7-1133-4900-93f6-dbf7fddeed5b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" Jul 23 00:27:21.887681 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.887505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/436060b9-c207-4a56-a70e-65e1a51c3b17-tmp\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.887681 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.887589 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf5354e7-1133-4900-93f6-dbf7fddeed5b-config\") pod \"service-ca-operator-5bcbc94c78-x8wz6\" (UID: \"bf5354e7-1133-4900-93f6-dbf7fddeed5b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" Jul 23 00:27:21.887681 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:21.887448 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:27:21.887681 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:21.887673 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls podName:c0835e02-d417-48ce-8bc4-0e28b009faf7 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:22.38765492 +0000 UTC m=+234.883323613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b4zzm" (UID: "c0835e02-d417-48ce-8bc4-0e28b009faf7") : secret "samples-operator-tls" not found Jul 23 00:27:21.887898 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.887870 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/436060b9-c207-4a56-a70e-65e1a51c3b17-tmp\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.888195 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.888175 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/436060b9-c207-4a56-a70e-65e1a51c3b17-service-ca-bundle\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.888253 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.888202 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/436060b9-c207-4a56-a70e-65e1a51c3b17-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.888365 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.888348 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/436060b9-c207-4a56-a70e-65e1a51c3b17-snapshots\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.889712 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.889697 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/436060b9-c207-4a56-a70e-65e1a51c3b17-serving-cert\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.889757 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.889709 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf5354e7-1133-4900-93f6-dbf7fddeed5b-serving-cert\") pod \"service-ca-operator-5bcbc94c78-x8wz6\" (UID: \"bf5354e7-1133-4900-93f6-dbf7fddeed5b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" Jul 23 00:27:21.895733 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.895713 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzhxw\" (UniqueName: \"kubernetes.io/projected/c0835e02-d417-48ce-8bc4-0e28b009faf7-kube-api-access-mzhxw\") pod \"cluster-samples-operator-79b7c55f87-b4zzm\" (UID: \"c0835e02-d417-48ce-8bc4-0e28b009faf7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" Jul 23 00:27:21.896269 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.896250 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d69lm\" (UniqueName: \"kubernetes.io/projected/436060b9-c207-4a56-a70e-65e1a51c3b17-kube-api-access-d69lm\") pod \"insights-operator-555d9d7885-wjhdx\" (UID: \"436060b9-c207-4a56-a70e-65e1a51c3b17\") " pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:21.896788 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.896766 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvt9k\" (UniqueName: \"kubernetes.io/projected/bf5354e7-1133-4900-93f6-dbf7fddeed5b-kube-api-access-rvt9k\") pod \"service-ca-operator-5bcbc94c78-x8wz6\" (UID: \"bf5354e7-1133-4900-93f6-dbf7fddeed5b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" Jul 23 00:27:21.896865 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.896839 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2k2l\" (UniqueName: \"kubernetes.io/projected/5068b739-0dd4-4654-a415-417a309565be-kube-api-access-w2k2l\") pod \"volume-data-source-validator-78cf4c6f6-h8gb6\" (UID: \"5068b739-0dd4-4654-a415-417a309565be\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-h8gb6" Jul 23 00:27:21.950890 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:21.950833 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:22.014209 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:22.014183 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:22.056326 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:22.056308 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-h8gb6" Jul 23 00:27:22.061854 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:22.061836 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" Jul 23 00:27:22.074641 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:22.074621 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-wjhdx" Jul 23 00:27:22.193289 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:22.193264 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-h8gb6"] Jul 23 00:27:22.196277 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:27:22.196254 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5068b739_0dd4_4654_a415_417a309565be.slice/crio-75402d30901f65dbb3505d8d84961927ef1e6388b9080df499976705cba41fed WatchSource:0}: Error finding container 75402d30901f65dbb3505d8d84961927ef1e6388b9080df499976705cba41fed: Status 404 returned error can't find the container with id 75402d30901f65dbb3505d8d84961927ef1e6388b9080df499976705cba41fed Jul 23 00:27:22.290674 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:22.290645 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:22.290801 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:22.290782 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:27:22.290846 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:22.290802 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77bf5fc84-r6bwx: secret "image-registry-tls" not found Jul 23 00:27:22.290882 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:22.290873 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls podName:42166b50-1d69-4b6f-a926-0a9f61f76c60 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:23.290859707 +0000 UTC m=+235.786528397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls") pod "image-registry-77bf5fc84-r6bwx" (UID: "42166b50-1d69-4b6f-a926-0a9f61f76c60") : secret "image-registry-tls" not found Jul 23 00:27:22.391430 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:22.391403 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b4zzm\" (UID: \"c0835e02-d417-48ce-8bc4-0e28b009faf7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" Jul 23 00:27:22.391575 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:22.391557 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:27:22.391630 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:22.391613 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls podName:c0835e02-d417-48ce-8bc4-0e28b009faf7 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:23.391600271 +0000 UTC m=+235.887268961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b4zzm" (UID: "c0835e02-d417-48ce-8bc4-0e28b009faf7") : secret "samples-operator-tls" not found Jul 23 00:27:22.416148 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:22.416078 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6"] Jul 23 00:27:22.418371 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:27:22.418348 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5354e7_1133_4900_93f6_dbf7fddeed5b.slice/crio-61c1fe5a456262f2bee2b69433848eab28a44b8da321652189df30e63ddeb80f WatchSource:0}: Error finding container 61c1fe5a456262f2bee2b69433848eab28a44b8da321652189df30e63ddeb80f: Status 404 returned error can't find the container with id 61c1fe5a456262f2bee2b69433848eab28a44b8da321652189df30e63ddeb80f Jul 23 00:27:22.418954 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:22.418932 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-wjhdx"] Jul 23 00:27:22.421988 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:27:22.421967 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436060b9_c207_4a56_a70e_65e1a51c3b17.slice/crio-855d6eca14c916bdd822818f8d6a40ad8807029667314d497d3c262528218492 WatchSource:0}: Error finding container 855d6eca14c916bdd822818f8d6a40ad8807029667314d497d3c262528218492: Status 404 returned error can't find the container with id 855d6eca14c916bdd822818f8d6a40ad8807029667314d497d3c262528218492 Jul 23 00:27:22.702804 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:22.702770 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-wjhdx" event={"ID":"436060b9-c207-4a56-a70e-65e1a51c3b17","Type":"ContainerStarted","Data":"855d6eca14c916bdd822818f8d6a40ad8807029667314d497d3c262528218492"} Jul 23 00:27:22.703742 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:22.703722 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" event={"ID":"bf5354e7-1133-4900-93f6-dbf7fddeed5b","Type":"ContainerStarted","Data":"61c1fe5a456262f2bee2b69433848eab28a44b8da321652189df30e63ddeb80f"} Jul 23 00:27:22.704658 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:22.704639 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-h8gb6" event={"ID":"5068b739-0dd4-4654-a415-417a309565be","Type":"ContainerStarted","Data":"75402d30901f65dbb3505d8d84961927ef1e6388b9080df499976705cba41fed"} Jul 23 00:27:23.299248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:23.299217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:23.299423 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:23.299328 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:27:23.299423 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:23.299340 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77bf5fc84-r6bwx: secret "image-registry-tls" not found Jul 23 00:27:23.299773 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:23.299732 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls podName:42166b50-1d69-4b6f-a926-0a9f61f76c60 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:25.29937588 +0000 UTC m=+237.795044570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls") pod "image-registry-77bf5fc84-r6bwx" (UID: "42166b50-1d69-4b6f-a926-0a9f61f76c60") : secret "image-registry-tls" not found Jul 23 00:27:23.400219 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:23.400182 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b4zzm\" (UID: \"c0835e02-d417-48ce-8bc4-0e28b009faf7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" Jul 23 00:27:23.400386 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:23.400367 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:27:23.400445 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:23.400432 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls podName:c0835e02-d417-48ce-8bc4-0e28b009faf7 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:25.400412968 +0000 UTC m=+237.896081659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b4zzm" (UID: "c0835e02-d417-48ce-8bc4-0e28b009faf7") : secret "samples-operator-tls" not found Jul 23 00:27:24.712961 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:24.712920 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-h8gb6" event={"ID":"5068b739-0dd4-4654-a415-417a309565be","Type":"ContainerStarted","Data":"eac858be235c57beee8540f9fee399d57d1d814af1d310e7d03d4d7f060f3295"} Jul 23 00:27:24.728075 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:24.728030 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-h8gb6" podStartSLOduration=2.0615581 podStartE2EDuration="3.728014508s" totalImagesPullingTime="1.666456408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:24.7274389 +0000 UTC m=+237.223107613" watchObservedRunningTime="2026-07-23 00:27:24.728014508 +0000 UTC m=+237.223683220" Jul 23 00:27:24.825673 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:24.825641 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:24.881749 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:24.881708 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:25.315989 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.315900 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:25.316124 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:25.316044 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:27:25.316124 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:25.316066 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77bf5fc84-r6bwx: secret "image-registry-tls" not found Jul 23 00:27:25.316202 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:25.316130 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls podName:42166b50-1d69-4b6f-a926-0a9f61f76c60 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:29.316106614 +0000 UTC m=+241.811775305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls") pod "image-registry-77bf5fc84-r6bwx" (UID: "42166b50-1d69-4b6f-a926-0a9f61f76c60") : secret "image-registry-tls" not found Jul 23 00:27:25.417215 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.417172 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b4zzm\" (UID: \"c0835e02-d417-48ce-8bc4-0e28b009faf7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" Jul 23 00:27:25.417388 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:25.417346 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:27:25.417445 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:25.417426 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls podName:c0835e02-d417-48ce-8bc4-0e28b009faf7 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:29.417406348 +0000 UTC m=+241.913075053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b4zzm" (UID: "c0835e02-d417-48ce-8bc4-0e28b009faf7") : secret "samples-operator-tls" not found Jul 23 00:27:25.576898 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.576828 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p2xg4"] Jul 23 00:27:25.579124 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.579094 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:25.583635 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.583615 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 00:27:25.583635 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.583623 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 00:27:25.584722 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.584705 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fm2gr\"" Jul 23 00:27:25.593336 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.593313 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p2xg4"] Jul 23 00:27:25.619004 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.618980 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-crio-socket\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:25.619105 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.619018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txj8h\" (UniqueName: \"kubernetes.io/projected/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-api-access-txj8h\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:25.619105 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.619062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-data-volume\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:25.619179 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.619134 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:25.619179 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.619163 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:25.626954 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.626928 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:25.686712 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.686687 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:25.717346 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.717321 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-wjhdx" event={"ID":"436060b9-c207-4a56-a70e-65e1a51c3b17","Type":"ContainerStarted","Data":"51bb1ed082e06a63af50eab7800d41985b9908bf080be5e7dc2db31a049ea5ba"} Jul 23 00:27:25.718804 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.718784 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" event={"ID":"bf5354e7-1133-4900-93f6-dbf7fddeed5b","Type":"ContainerStarted","Data":"fc42627d3ac0e663bc0db2411bfdff975118ec727899870a4e978d4111b941b0"} Jul 23 00:27:25.720183 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.720164 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-crio-socket\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:25.720251 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.720198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txj8h\" (UniqueName: \"kubernetes.io/projected/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-api-access-txj8h\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:25.720251 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.720227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-data-volume\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:25.720364 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.720275 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-crio-socket\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:25.720422 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.720403 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:25.720474 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:25.720462 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:27:25.720522 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:25.720502 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:27:25.720522 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.720460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:25.720522 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:25.720509 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls podName:cc4dfee5-f1cd-4494-90ca-c68bfef5f363 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:26.220492702 +0000 UTC m=+238.716161394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2xg4" (UID: "cc4dfee5-f1cd-4494-90ca-c68bfef5f363") : secret "insights-runtime-extractor-tls" not found Jul 23 00:27:25.720668 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:25.720569 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm podName:cc4dfee5-f1cd-4494-90ca-c68bfef5f363 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:26.220553492 +0000 UTC m=+238.716222203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2xg4" (UID: "cc4dfee5-f1cd-4494-90ca-c68bfef5f363") : configmap "kube-rbac-proxy" not found Jul 23 00:27:25.720668 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.720587 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-data-volume\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:25.728833 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.728817 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txj8h\" (UniqueName: \"kubernetes.io/projected/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-api-access-txj8h\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:25.735567 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.735514 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-wjhdx" podStartSLOduration=2.194259795 podStartE2EDuration="4.735500485s" totalImagesPullingTime="2.54124069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:25.735115536 +0000 UTC m=+238.230784247" watchObservedRunningTime="2026-07-23 00:27:25.735500485 +0000 UTC m=+238.231169196" Jul 23 00:27:25.749374 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:25.749337 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" podStartSLOduration=2.2050213579999998 podStartE2EDuration="4.749324485s" totalImagesPullingTime="2.544303127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:25.748641617 +0000 UTC m=+238.244310342" watchObservedRunningTime="2026-07-23 00:27:25.749324485 +0000 UTC m=+238.244993248" Jul 23 00:27:26.224941 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.224912 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:26.225081 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.224953 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:26.225081 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:26.225028 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:27:26.225081 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:26.225057 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:27:26.225193 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:26.225089 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm podName:cc4dfee5-f1cd-4494-90ca-c68bfef5f363 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:27.225071362 +0000 UTC m=+239.720740067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2xg4" (UID: "cc4dfee5-f1cd-4494-90ca-c68bfef5f363") : configmap "kube-rbac-proxy" not found Jul 23 00:27:26.225193 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:26.225106 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls podName:cc4dfee5-f1cd-4494-90ca-c68bfef5f363 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:27.225097346 +0000 UTC m=+239.720766037 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2xg4" (UID: "cc4dfee5-f1cd-4494-90ca-c68bfef5f363") : secret "insights-runtime-extractor-tls" not found Jul 23 00:27:26.461978 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.461941 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:26.485732 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.485673 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwk9s"] Jul 23 00:27:26.487968 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.487953 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwk9s" Jul 23 00:27:26.490619 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.490598 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 23 00:27:26.491829 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.491806 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7scjz\"" Jul 23 00:27:26.491829 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.491824 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 23 00:27:26.500546 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.499151 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwk9s"] Jul 23 00:27:26.526759 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.526733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wrt7\" (UniqueName: \"kubernetes.io/projected/5b4cf7c3-4507-4a93-a2ca-7a4ced835de5-kube-api-access-5wrt7\") pod \"migrator-7cc9c7ffd7-cwk9s\" (UID: \"5b4cf7c3-4507-4a93-a2ca-7a4ced835de5\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwk9s" Jul 23 00:27:26.527273 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.527246 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:26.589678 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.589649 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:26.627739 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.627717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wrt7\" (UniqueName: \"kubernetes.io/projected/5b4cf7c3-4507-4a93-a2ca-7a4ced835de5-kube-api-access-5wrt7\") pod \"migrator-7cc9c7ffd7-cwk9s\" (UID: \"5b4cf7c3-4507-4a93-a2ca-7a4ced835de5\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwk9s" Jul 23 00:27:26.635379 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.635357 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wrt7\" (UniqueName: \"kubernetes.io/projected/5b4cf7c3-4507-4a93-a2ca-7a4ced835de5-kube-api-access-5wrt7\") pod \"migrator-7cc9c7ffd7-cwk9s\" (UID: \"5b4cf7c3-4507-4a93-a2ca-7a4ced835de5\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwk9s" Jul 23 00:27:26.796490 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.796420 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwk9s" Jul 23 00:27:26.912744 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:26.912582 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwk9s"] Jul 23 00:27:26.915083 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:27:26.915051 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4cf7c3_4507_4a93_a2ca_7a4ced835de5.slice/crio-81a40c97be95cabbc666ad8a17295bdbefee4202e3d216f7971bbe546b92eef6 WatchSource:0}: Error finding container 81a40c97be95cabbc666ad8a17295bdbefee4202e3d216f7971bbe546b92eef6: Status 404 returned error can't find the container with id 81a40c97be95cabbc666ad8a17295bdbefee4202e3d216f7971bbe546b92eef6 Jul 23 00:27:27.028948 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:27.028917 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:27.080520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:27.080457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:27.143435 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:27.143409 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:27.210985 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:27.210951 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:27.233795 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:27.233775 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:27.233883 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:27.233818 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:27.233916 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:27.233903 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:27:27.233916 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:27.233908 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:27:27.233983 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:27.233953 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm podName:cc4dfee5-f1cd-4494-90ca-c68bfef5f363 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:29.233940444 +0000 UTC m=+241.729609134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2xg4" (UID: "cc4dfee5-f1cd-4494-90ca-c68bfef5f363") : configmap "kube-rbac-proxy" not found Jul 23 00:27:27.233983 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:27.233965 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls podName:cc4dfee5-f1cd-4494-90ca-c68bfef5f363 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:29.233959535 +0000 UTC m=+241.729628225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2xg4" (UID: "cc4dfee5-f1cd-4494-90ca-c68bfef5f363") : secret "insights-runtime-extractor-tls" not found Jul 23 00:27:27.630394 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:27.630358 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:27.726115 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:27.726083 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwk9s" event={"ID":"5b4cf7c3-4507-4a93-a2ca-7a4ced835de5","Type":"ContainerStarted","Data":"81a40c97be95cabbc666ad8a17295bdbefee4202e3d216f7971bbe546b92eef6"} Jul 23 00:27:27.730042 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:27.730013 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:27.864861 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:27.864836 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jl4hg_91698027-ce8c-4729-86e9-6eb62632951d/dns-node-resolver/0.log" Jul 23 00:27:28.730182 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:28.730145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwk9s" event={"ID":"5b4cf7c3-4507-4a93-a2ca-7a4ced835de5","Type":"ContainerStarted","Data":"8e2c4b969e66826157607e7e5ade18cd31f40f0b7b70924b5e176bab5e8e468d"} Jul 23 00:27:28.730370 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:28.730237 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwk9s" event={"ID":"5b4cf7c3-4507-4a93-a2ca-7a4ced835de5","Type":"ContainerStarted","Data":"57a2e19147c47398760e34e6d444c504af8f68c3541aedcd221c55adcf370e31"} Jul 23 00:27:28.746362 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:28.746314 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-cwk9s" podStartSLOduration=1.361922965 podStartE2EDuration="2.746297539s" totalImagesPullingTime="1.384374574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:28.744987713 +0000 UTC m=+241.240656426" watchObservedRunningTime="2026-07-23 00:27:28.746297539 +0000 UTC m=+241.241966251" Jul 23 00:27:28.820227 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:28.820204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:28.864390 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:28.864371 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zxnfn_c0e82032-5917-427c-b7b6-e7597a1390d3/node-ca/0.log" Jul 23 00:27:29.210958 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.210907 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-tg2kp"] Jul 23 00:27:29.212986 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.212967 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-tg2kp" Jul 23 00:27:29.215494 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.215468 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 23 00:27:29.215829 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.215804 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 23 00:27:29.215925 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.215842 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 23 00:27:29.216925 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.216896 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lm45c\"" Jul 23 00:27:29.217030 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.216931 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 23 00:27:29.220769 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.220752 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-tg2kp"] Jul 23 00:27:29.246017 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.245992 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:29.247845 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.247828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:29.247911 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.247869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:29.247953 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:29.247940 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:27:29.247990 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:29.247959 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:27:29.248021 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:29.247991 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm podName:cc4dfee5-f1cd-4494-90ca-c68bfef5f363 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:33.24798004 +0000 UTC m=+245.743648730 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2xg4" (UID: "cc4dfee5-f1cd-4494-90ca-c68bfef5f363") : configmap "kube-rbac-proxy" not found Jul 23 00:27:29.248021 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:29.248003 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls podName:cc4dfee5-f1cd-4494-90ca-c68bfef5f363 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:33.247997141 +0000 UTC m=+245.743665831 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2xg4" (UID: "cc4dfee5-f1cd-4494-90ca-c68bfef5f363") : secret "insights-runtime-extractor-tls" not found Jul 23 00:27:29.264629 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.264612 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kdnq4_e5461539-d692-499b-9b64-4d5b6def6be5/serve-healthcheck-canary/0.log" Jul 23 00:27:29.304746 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.304719 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:29.349040 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.349017 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b166c651-8c6c-4f65-ba94-933437d3658e-signing-cabundle\") pod \"service-ca-58598695f7-tg2kp\" (UID: \"b166c651-8c6c-4f65-ba94-933437d3658e\") " pod="openshift-service-ca/service-ca-58598695f7-tg2kp" Jul 23 00:27:29.349107 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.349065 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:29.349107 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.349098 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnltt\" (UniqueName: \"kubernetes.io/projected/b166c651-8c6c-4f65-ba94-933437d3658e-kube-api-access-nnltt\") pod \"service-ca-58598695f7-tg2kp\" (UID: \"b166c651-8c6c-4f65-ba94-933437d3658e\") " pod="openshift-service-ca/service-ca-58598695f7-tg2kp" Jul 23 00:27:29.349187 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.349127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b166c651-8c6c-4f65-ba94-933437d3658e-signing-key\") pod \"service-ca-58598695f7-tg2kp\" (UID: \"b166c651-8c6c-4f65-ba94-933437d3658e\") " pod="openshift-service-ca/service-ca-58598695f7-tg2kp" Jul 23 00:27:29.349187 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:29.349149 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 00:27:29.349187 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:29.349162 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77bf5fc84-r6bwx: secret "image-registry-tls" not found Jul 23 00:27:29.349283 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:29.349198 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls podName:42166b50-1d69-4b6f-a926-0a9f61f76c60 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:37.349184912 +0000 UTC m=+249.844853602 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls") pod "image-registry-77bf5fc84-r6bwx" (UID: "42166b50-1d69-4b6f-a926-0a9f61f76c60") : secret "image-registry-tls" not found Jul 23 00:27:29.450300 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.450273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b166c651-8c6c-4f65-ba94-933437d3658e-signing-key\") pod \"service-ca-58598695f7-tg2kp\" (UID: \"b166c651-8c6c-4f65-ba94-933437d3658e\") " pod="openshift-service-ca/service-ca-58598695f7-tg2kp" Jul 23 00:27:29.450394 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.450307 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b4zzm\" (UID: \"c0835e02-d417-48ce-8bc4-0e28b009faf7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" Jul 23 00:27:29.450394 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.450358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b166c651-8c6c-4f65-ba94-933437d3658e-signing-cabundle\") pod \"service-ca-58598695f7-tg2kp\" (UID: \"b166c651-8c6c-4f65-ba94-933437d3658e\") " pod="openshift-service-ca/service-ca-58598695f7-tg2kp" Jul 23 00:27:29.450394 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.450390 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnltt\" (UniqueName: \"kubernetes.io/projected/b166c651-8c6c-4f65-ba94-933437d3658e-kube-api-access-nnltt\") pod \"service-ca-58598695f7-tg2kp\" (UID: \"b166c651-8c6c-4f65-ba94-933437d3658e\") " pod="openshift-service-ca/service-ca-58598695f7-tg2kp" Jul 23 00:27:29.450572 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:29.450555 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 00:27:29.450630 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:29.450619 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls podName:c0835e02-d417-48ce-8bc4-0e28b009faf7 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:37.450602768 +0000 UTC m=+249.946271479 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-b4zzm" (UID: "c0835e02-d417-48ce-8bc4-0e28b009faf7") : secret "samples-operator-tls" not found Jul 23 00:27:29.450985 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.450968 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b166c651-8c6c-4f65-ba94-933437d3658e-signing-cabundle\") pod \"service-ca-58598695f7-tg2kp\" (UID: \"b166c651-8c6c-4f65-ba94-933437d3658e\") " pod="openshift-service-ca/service-ca-58598695f7-tg2kp" Jul 23 00:27:29.452641 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.452623 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b166c651-8c6c-4f65-ba94-933437d3658e-signing-key\") pod \"service-ca-58598695f7-tg2kp\" (UID: \"b166c651-8c6c-4f65-ba94-933437d3658e\") " pod="openshift-service-ca/service-ca-58598695f7-tg2kp" Jul 23 00:27:29.459051 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.459033 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnltt\" (UniqueName: \"kubernetes.io/projected/b166c651-8c6c-4f65-ba94-933437d3658e-kube-api-access-nnltt\") pod \"service-ca-58598695f7-tg2kp\" (UID: \"b166c651-8c6c-4f65-ba94-933437d3658e\") " pod="openshift-service-ca/service-ca-58598695f7-tg2kp" Jul 23 00:27:29.522399 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.522347 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-tg2kp" Jul 23 00:27:29.640816 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.640787 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-tg2kp"] Jul 23 00:27:29.644879 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:27:29.644854 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb166c651_8c6c_4f65_ba94_933437d3658e.slice/crio-aeee2a3bf569fa82645a57db69695505eac839f0718ef2d63e1ab53d81cba5a4 WatchSource:0}: Error finding container aeee2a3bf569fa82645a57db69695505eac839f0718ef2d63e1ab53d81cba5a4: Status 404 returned error can't find the container with id aeee2a3bf569fa82645a57db69695505eac839f0718ef2d63e1ab53d81cba5a4 Jul 23 00:27:29.734762 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.734736 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-tg2kp" event={"ID":"b166c651-8c6c-4f65-ba94-933437d3658e","Type":"ContainerStarted","Data":"450875b23884160a125cc9542cc8ca536f6413242e01906b3113307d7fb05c70"} Jul 23 00:27:29.734874 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.734782 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-tg2kp" event={"ID":"b166c651-8c6c-4f65-ba94-933437d3658e","Type":"ContainerStarted","Data":"aeee2a3bf569fa82645a57db69695505eac839f0718ef2d63e1ab53d81cba5a4"} Jul 23 00:27:29.755059 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:29.755017 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-tg2kp" podStartSLOduration=0.755003252 podStartE2EDuration="755.003252ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:29.754388501 +0000 UTC m=+242.250057212" watchObservedRunningTime="2026-07-23 00:27:29.755003252 +0000 UTC m=+242.250671964" Jul 23 00:27:30.089744 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:30.089714 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:31.531908 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:31.531869 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:31.771840 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:31.771796 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:31.951432 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:31.951399 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:32.042686 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:32.042641 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:32.231514 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:32.231428 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:32.990821 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:32.990792 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:33.090783 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:33.090753 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:33.226260 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:33.226233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:33.282045 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:33.281978 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:33.282045 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:33.282018 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:33.282224 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:33.282117 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 00:27:33.282224 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:33.282140 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:27:33.282224 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:33.282180 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm podName:cc4dfee5-f1cd-4494-90ca-c68bfef5f363 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:41.282156114 +0000 UTC m=+253.777824820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2xg4" (UID: "cc4dfee5-f1cd-4494-90ca-c68bfef5f363") : configmap "kube-rbac-proxy" not found Jul 23 00:27:33.282224 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:33.282193 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls podName:cc4dfee5-f1cd-4494-90ca-c68bfef5f363 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:41.282187154 +0000 UTC m=+253.777855845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2xg4" (UID: "cc4dfee5-f1cd-4494-90ca-c68bfef5f363") : secret "insights-runtime-extractor-tls" not found Jul 23 00:27:33.628909 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:33.628827 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:34.234543 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:34.234477 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:34.629302 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:34.629227 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:34.832120 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:34.832089 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:35.237999 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:35.237969 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:35.425055 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:35.425024 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:35.632098 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:35.632020 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:35.839966 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:35.839939 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:36.219116 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:36.219085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:36.443904 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:36.443865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:37.024341 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:37.024308 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:37.412807 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:37.412759 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:37.415063 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:37.415032 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls\") pod \"image-registry-77bf5fc84-r6bwx\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:37.422218 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:37.422191 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:37.514093 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:37.514074 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b4zzm\" (UID: \"c0835e02-d417-48ce-8bc4-0e28b009faf7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" Jul 23 00:27:37.516547 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:37.516505 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0835e02-d417-48ce-8bc4-0e28b009faf7-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-b4zzm\" (UID: \"c0835e02-d417-48ce-8bc4-0e28b009faf7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" Jul 23 00:27:37.563583 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:37.563562 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6v7ht\"" Jul 23 00:27:37.571896 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:37.571877 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:37.670762 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:37.670698 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tj2kx\"" Jul 23 00:27:37.678891 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:37.678871 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" Jul 23 00:27:37.692151 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:37.692129 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77bf5fc84-r6bwx"] Jul 23 00:27:37.695218 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:27:37.695197 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42166b50_1d69_4b6f_a926_0a9f61f76c60.slice/crio-17f213c1b8b4ed063f60982686a1e0b88d6fef557e0c6732eed9199772013355 WatchSource:0}: Error finding container 17f213c1b8b4ed063f60982686a1e0b88d6fef557e0c6732eed9199772013355: Status 404 returned error can't find the container with id 17f213c1b8b4ed063f60982686a1e0b88d6fef557e0c6732eed9199772013355 Jul 23 00:27:37.760166 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:37.759887 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" event={"ID":"42166b50-1d69-4b6f-a926-0a9f61f76c60","Type":"ContainerStarted","Data":"17f213c1b8b4ed063f60982686a1e0b88d6fef557e0c6732eed9199772013355"} Jul 23 00:27:37.806259 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:37.806236 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm"] Jul 23 00:27:37.822145 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:37.822098 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:38.023843 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:38.023765 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:38.075294 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:38.075266 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:38.622934 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:38.622887 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:38.764358 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:38.764330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" event={"ID":"42166b50-1d69-4b6f-a926-0a9f61f76c60","Type":"ContainerStarted","Data":"80da79d18ddcc478d608e462fe9fc40c1ca40ff1b20574cb5909ce5b678d520f"} Jul 23 00:27:38.764560 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:38.764463 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:27:38.765423 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:38.765398 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" event={"ID":"c0835e02-d417-48ce-8bc4-0e28b009faf7","Type":"ContainerStarted","Data":"737b444263c5206f34decceb97c3d91b5288578288d1db611fe26cdb98d29eba"} Jul 23 00:27:38.783100 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:38.783057 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" podStartSLOduration=17.783043215 podStartE2EDuration="17.783043215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:38.782053019 +0000 UTC m=+251.277721755" watchObservedRunningTime="2026-07-23 00:27:38.783043215 +0000 UTC m=+251.278711926" Jul 23 00:27:38.821300 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:38.821265 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:39.227514 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:39.227481 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:39.426591 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:39.426553 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:39.622196 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:39.622076 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:39.822754 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:39.822703 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:39.932749 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:39.932720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs\") pod \"network-metrics-daemon-mx6kd\" (UID: \"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246\") " pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:27:39.935443 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:39.935408 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246-metrics-certs\") pod \"network-metrics-daemon-mx6kd\" (UID: \"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246\") " pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:27:39.971278 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:39.971259 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8gvmb\"" Jul 23 00:27:39.978690 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:39.978668 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mx6kd" Jul 23 00:27:40.119610 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:40.119581 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mx6kd"] Jul 23 00:27:40.122686 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:27:40.122650 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a38f4c1_017f_4c0c_b9c2_4e60e9f2e246.slice/crio-974192e5a71e543f9102252811ea95431893b666002c0b5dd34a0d72fbfa41b3 WatchSource:0}: Error finding container 974192e5a71e543f9102252811ea95431893b666002c0b5dd34a0d72fbfa41b3: Status 404 returned error can't find the container with id 974192e5a71e543f9102252811ea95431893b666002c0b5dd34a0d72fbfa41b3 Jul 23 00:27:40.629872 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:40.629839 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:40.774417 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:40.774387 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" event={"ID":"c0835e02-d417-48ce-8bc4-0e28b009faf7","Type":"ContainerStarted","Data":"209241afdf2b436572213484006a524ec056454821292cb4dadc07431d8f0cc3"} Jul 23 00:27:40.774589 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:40.774440 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" event={"ID":"c0835e02-d417-48ce-8bc4-0e28b009faf7","Type":"ContainerStarted","Data":"25931cd73f72548874d98be0703554a54615ea1dac2b7607eb90beb5b0dbaae9"} Jul 23 00:27:40.775481 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:40.775449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mx6kd" event={"ID":"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246","Type":"ContainerStarted","Data":"974192e5a71e543f9102252811ea95431893b666002c0b5dd34a0d72fbfa41b3"} Jul 23 00:27:40.792100 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:40.792050 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-b4zzm" podStartSLOduration=17.729096111 podStartE2EDuration="19.792033452s" totalImagesPullingTime="2.062937341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:40.790872928 +0000 UTC m=+253.286541640" watchObservedRunningTime="2026-07-23 00:27:40.792033452 +0000 UTC m=+253.287702165" Jul 23 00:27:40.844090 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:40.844058 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:41.218488 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:41.218450 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:41.346423 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:41.346388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:41.346596 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:41.346446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:41.346596 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:41.346557 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 00:27:41.346722 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:27:41.346632 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls podName:cc4dfee5-f1cd-4494-90ca-c68bfef5f363 nodeName:}" failed. No retries permitted until 2026-07-23 00:27:57.34661246 +0000 UTC m=+269.842281164 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2xg4" (UID: "cc4dfee5-f1cd-4494-90ca-c68bfef5f363") : secret "insights-runtime-extractor-tls" not found Jul 23 00:27:41.346988 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:41.346963 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:41.359904 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:41.359881 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:41.620673 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:41.620641 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:41.781128 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:41.781058 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mx6kd" event={"ID":"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246","Type":"ContainerStarted","Data":"8cbb431dddab464f39abef870b141b5289864bcf1daf177eb74e5778695c5690"} Jul 23 00:27:41.781128 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:41.781114 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mx6kd" event={"ID":"9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246","Type":"ContainerStarted","Data":"9b1714e425a3bc3b35a7d70f298115e779c084e8ce8f5fa606684ceaa644cc70"} Jul 23 00:27:41.801922 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:41.801876 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mx6kd" podStartSLOduration=251.577586389 podStartE2EDuration="4m12.801856734s" totalImagesPullingTime="1.224270345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:27:41.80037736 +0000 UTC m=+254.296046072" watchObservedRunningTime="2026-07-23 00:27:41.801856734 +0000 UTC m=+254.297525449" Jul 23 00:27:42.027744 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:42.027707 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:42.221286 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:42.221254 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:49.406947 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:49.406917 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:50.612540 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:50.612494 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:51.758229 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:51.758194 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77bf5fc84-r6bwx"] Jul 23 00:27:51.806199 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:51.806166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:51.890746 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:51.890711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:56.294402 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:56.294363 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:56.393926 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:56.393895 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:56.469188 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:56.469156 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:56.984504 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:56.984465 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:27:57.360172 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:57.360090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:57.362328 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:57.362308 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cc4dfee5-f1cd-4494-90ca-c68bfef5f363-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2xg4\" (UID: \"cc4dfee5-f1cd-4494-90ca-c68bfef5f363\") " pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:57.391925 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:57.391900 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fm2gr\"" Jul 23 00:27:57.400047 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:57.400031 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2xg4" Jul 23 00:27:57.519913 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:57.519879 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p2xg4"] Jul 23 00:27:57.524580 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:27:57.524550 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4dfee5_f1cd_4494_90ca_c68bfef5f363.slice/crio-7695a27a94d72ea35d1eeb5e4740d31b4b40e7305f0f4daf0f8d08a1b26281bd WatchSource:0}: Error finding container 7695a27a94d72ea35d1eeb5e4740d31b4b40e7305f0f4daf0f8d08a1b26281bd: Status 404 returned error can't find the container with id 7695a27a94d72ea35d1eeb5e4740d31b4b40e7305f0f4daf0f8d08a1b26281bd Jul 23 00:27:57.827273 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:27:57.827240 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2xg4" event={"ID":"cc4dfee5-f1cd-4494-90ca-c68bfef5f363","Type":"ContainerStarted","Data":"7695a27a94d72ea35d1eeb5e4740d31b4b40e7305f0f4daf0f8d08a1b26281bd"} Jul 23 00:28:00.481976 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:00.481937 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:00.562642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:00.562605 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:00.675998 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:00.675960 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:01.764209 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:01.764170 2564 patch_prober.go:28] interesting pod/image-registry-77bf5fc84-r6bwx container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 23 00:28:01.764612 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:01.764236 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" podUID="42166b50-1d69-4b6f-a926-0a9f61f76c60" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 00:28:02.850739 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:02.850706 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2xg4" event={"ID":"cc4dfee5-f1cd-4494-90ca-c68bfef5f363","Type":"ContainerStarted","Data":"5608eb2cc54ecd16fffe52051e78bac9c5e2a8ab7070ed90964916f49d739042"} Jul 23 00:28:06.466990 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:06.466953 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:06.868642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:06.868556 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2xg4" event={"ID":"cc4dfee5-f1cd-4494-90ca-c68bfef5f363","Type":"ContainerStarted","Data":"70eeeb166269dea6cb3b62679587f4e59d5fcde15ad28c468705431e956dc4b7"} Jul 23 00:28:06.889119 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:06.889080 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:06.946239 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:06.946199 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:08.603385 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.603347 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:08.818556 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.818483 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:08.845180 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.844743 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6"] Jul 23 00:28:08.864591 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.862734 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-jm5s2"] Jul 23 00:28:08.866315 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.865207 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" Jul 23 00:28:08.869084 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.868430 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 00:28:08.869084 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.868484 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 00:28:08.869084 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.868434 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 23 00:28:08.869084 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.868715 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 00:28:08.869084 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.868644 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 23 00:28:08.869084 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.868944 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9txqv\"" Jul 23 00:28:08.870263 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.870237 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6"] Jul 23 00:28:08.870356 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.870285 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r8pxs"] Jul 23 00:28:08.874000 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.870505 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:08.874000 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.873975 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 23 00:28:08.876455 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.874247 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 23 00:28:08.876455 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.874314 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 23 00:28:08.876455 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.874443 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ffh8d\"" Jul 23 00:28:08.876455 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.874752 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:08.876455 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.876378 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-jm5s2"] Jul 23 00:28:08.879171 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.879127 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 00:28:08.879171 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.879138 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kwh25\"" Jul 23 00:28:08.880939 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.879346 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 00:28:08.880939 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.879409 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 00:28:08.903724 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.903429 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:08.955315 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0546d769-8f4f-4c68-be13-64bab60debb1-metrics-client-ca\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:08.955315 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955325 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/066d1d20-c2c9-4a0f-8979-bd68e53154b7-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:08.955580 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-textfile\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:08.955580 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-wtmp\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:08.955580 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955411 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/066d1d20-c2c9-4a0f-8979-bd68e53154b7-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:08.955580 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955441 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddac8ecd-70f8-498d-a8a9-eb9ff2278241-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-cj7k6\" (UID: \"ddac8ecd-70f8-498d-a8a9-eb9ff2278241\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" Jul 23 00:28:08.955580 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955468 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddac8ecd-70f8-498d-a8a9-eb9ff2278241-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-cj7k6\" (UID: \"ddac8ecd-70f8-498d-a8a9-eb9ff2278241\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" Jul 23 00:28:08.955580 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955492 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6hpz\" (UniqueName: \"kubernetes.io/projected/066d1d20-c2c9-4a0f-8979-bd68e53154b7-kube-api-access-p6hpz\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:08.955580 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0546d769-8f4f-4c68-be13-64bab60debb1-root\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:08.955580 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955570 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/066d1d20-c2c9-4a0f-8979-bd68e53154b7-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:08.956001 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955598 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55q86\" (UniqueName: \"kubernetes.io/projected/0546d769-8f4f-4c68-be13-64bab60debb1-kube-api-access-55q86\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:08.956001 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955639 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0546d769-8f4f-4c68-be13-64bab60debb1-sys\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:08.956001 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955661 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:08.956001 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955686 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddac8ecd-70f8-498d-a8a9-eb9ff2278241-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-cj7k6\" (UID: \"ddac8ecd-70f8-498d-a8a9-eb9ff2278241\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" Jul 23 00:28:08.956001 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-accelerators-collector-config\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:08.956001 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955742 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-tls\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:08.956001 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955783 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/066d1d20-c2c9-4a0f-8979-bd68e53154b7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:08.956001 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlh6n\" (UniqueName: \"kubernetes.io/projected/ddac8ecd-70f8-498d-a8a9-eb9ff2278241-kube-api-access-jlh6n\") pod \"openshift-state-metrics-79f6958cdf-cj7k6\" (UID: \"ddac8ecd-70f8-498d-a8a9-eb9ff2278241\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" Jul 23 00:28:08.956001 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:08.955833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/066d1d20-c2c9-4a0f-8979-bd68e53154b7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:09.057111 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057072 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0546d769-8f4f-4c68-be13-64bab60debb1-root\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.057298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/066d1d20-c2c9-4a0f-8979-bd68e53154b7-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:09.057298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057196 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0546d769-8f4f-4c68-be13-64bab60debb1-root\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.057298 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:28:09.057233 2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 23 00:28:09.057298 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55q86\" (UniqueName: \"kubernetes.io/projected/0546d769-8f4f-4c68-be13-64bab60debb1-kube-api-access-55q86\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.057508 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0546d769-8f4f-4c68-be13-64bab60debb1-sys\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.057508 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057331 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.057508 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddac8ecd-70f8-498d-a8a9-eb9ff2278241-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-cj7k6\" (UID: \"ddac8ecd-70f8-498d-a8a9-eb9ff2278241\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" Jul 23 00:28:09.057508 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057393 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-accelerators-collector-config\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.057508 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057421 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-tls\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.057508 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/066d1d20-c2c9-4a0f-8979-bd68e53154b7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:09.057508 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057490 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlh6n\" (UniqueName: \"kubernetes.io/projected/ddac8ecd-70f8-498d-a8a9-eb9ff2278241-kube-api-access-jlh6n\") pod \"openshift-state-metrics-79f6958cdf-cj7k6\" (UID: \"ddac8ecd-70f8-498d-a8a9-eb9ff2278241\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" Jul 23 00:28:09.057878 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/066d1d20-c2c9-4a0f-8979-bd68e53154b7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:09.057878 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057572 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0546d769-8f4f-4c68-be13-64bab60debb1-metrics-client-ca\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.057878 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057621 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/066d1d20-c2c9-4a0f-8979-bd68e53154b7-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:09.057878 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:28:09.057629 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 23 00:28:09.057878 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057656 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-textfile\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.057878 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:28:09.057677 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-tls podName:0546d769-8f4f-4c68-be13-64bab60debb1 nodeName:}" failed. No retries permitted until 2026-07-23 00:28:09.557657711 +0000 UTC m=+282.053326408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-tls") pod "node-exporter-r8pxs" (UID: "0546d769-8f4f-4c68-be13-64bab60debb1") : secret "node-exporter-tls" not found Jul 23 00:28:09.057878 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-wtmp\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.057878 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/066d1d20-c2c9-4a0f-8979-bd68e53154b7-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:09.057878 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddac8ecd-70f8-498d-a8a9-eb9ff2278241-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-cj7k6\" (UID: \"ddac8ecd-70f8-498d-a8a9-eb9ff2278241\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" Jul 23 00:28:09.057878 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddac8ecd-70f8-498d-a8a9-eb9ff2278241-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-cj7k6\" (UID: \"ddac8ecd-70f8-498d-a8a9-eb9ff2278241\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" Jul 23 00:28:09.057878 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6hpz\" (UniqueName: \"kubernetes.io/projected/066d1d20-c2c9-4a0f-8979-bd68e53154b7-kube-api-access-p6hpz\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:09.058415 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.057967 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-textfile\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.058415 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.058000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-wtmp\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.058515 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.058483 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-accelerators-collector-config\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.059367 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:28:09.058593 2564 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 23 00:28:09.059367 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:28:09.058650 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddac8ecd-70f8-498d-a8a9-eb9ff2278241-openshift-state-metrics-tls podName:ddac8ecd-70f8-498d-a8a9-eb9ff2278241 nodeName:}" failed. No retries permitted until 2026-07-23 00:28:09.55863578 +0000 UTC m=+282.054304474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ddac8ecd-70f8-498d-a8a9-eb9ff2278241-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-cj7k6" (UID: "ddac8ecd-70f8-498d-a8a9-eb9ff2278241") : secret "openshift-state-metrics-tls" not found Jul 23 00:28:09.059367 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.058828 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/066d1d20-c2c9-4a0f-8979-bd68e53154b7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:09.059367 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.059050 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/066d1d20-c2c9-4a0f-8979-bd68e53154b7-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:09.059367 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.059250 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/066d1d20-c2c9-4a0f-8979-bd68e53154b7-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:09.059367 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.059263 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddac8ecd-70f8-498d-a8a9-eb9ff2278241-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-cj7k6\" (UID: \"ddac8ecd-70f8-498d-a8a9-eb9ff2278241\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" Jul 23 00:28:09.059367 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.059287 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0546d769-8f4f-4c68-be13-64bab60debb1-metrics-client-ca\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.059367 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.059311 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0546d769-8f4f-4c68-be13-64bab60debb1-sys\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.059367 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:28:09.059318 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066d1d20-c2c9-4a0f-8979-bd68e53154b7-kube-state-metrics-tls podName:066d1d20-c2c9-4a0f-8979-bd68e53154b7 nodeName:}" failed. No retries permitted until 2026-07-23 00:28:09.55930365 +0000 UTC m=+282.054972351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/066d1d20-c2c9-4a0f-8979-bd68e53154b7-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-jm5s2" (UID: "066d1d20-c2c9-4a0f-8979-bd68e53154b7") : secret "kube-state-metrics-tls" not found Jul 23 00:28:09.060789 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.060764 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddac8ecd-70f8-498d-a8a9-eb9ff2278241-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-cj7k6\" (UID: \"ddac8ecd-70f8-498d-a8a9-eb9ff2278241\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" Jul 23 00:28:09.061839 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.061795 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.062740 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.062719 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/066d1d20-c2c9-4a0f-8979-bd68e53154b7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:09.069914 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.069881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6hpz\" (UniqueName: \"kubernetes.io/projected/066d1d20-c2c9-4a0f-8979-bd68e53154b7-kube-api-access-p6hpz\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:09.070704 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.070681 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55q86\" (UniqueName: \"kubernetes.io/projected/0546d769-8f4f-4c68-be13-64bab60debb1-kube-api-access-55q86\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.072078 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.072035 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlh6n\" (UniqueName: \"kubernetes.io/projected/ddac8ecd-70f8-498d-a8a9-eb9ff2278241-kube-api-access-jlh6n\") pod \"openshift-state-metrics-79f6958cdf-cj7k6\" (UID: \"ddac8ecd-70f8-498d-a8a9-eb9ff2278241\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" Jul 23 00:28:09.105102 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.105061 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:09.205762 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.205643 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:09.309184 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.309130 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:09.406974 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.406934 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:09.528555 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:28:09.528444 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-2gcnv" podUID="cba4baf3-9d99-4bb1-a822-8d80d49c1f9a" Jul 23 00:28:09.547761 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.547711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:09.561592 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.561558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/066d1d20-c2c9-4a0f-8979-bd68e53154b7-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:09.561736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.561605 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-tls\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.561736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.561661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddac8ecd-70f8-498d-a8a9-eb9ff2278241-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-cj7k6\" (UID: \"ddac8ecd-70f8-498d-a8a9-eb9ff2278241\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" Jul 23 00:28:09.564033 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.564000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ddac8ecd-70f8-498d-a8a9-eb9ff2278241-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-cj7k6\" (UID: \"ddac8ecd-70f8-498d-a8a9-eb9ff2278241\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" Jul 23 00:28:09.564148 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.564132 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/066d1d20-c2c9-4a0f-8979-bd68e53154b7-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jm5s2\" (UID: \"066d1d20-c2c9-4a0f-8979-bd68e53154b7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:09.564228 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.564209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0546d769-8f4f-4c68-be13-64bab60debb1-node-exporter-tls\") pod \"node-exporter-r8pxs\" (UID: \"0546d769-8f4f-4c68-be13-64bab60debb1\") " pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.784453 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.784385 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" Jul 23 00:28:09.798154 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.798128 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" Jul 23 00:28:09.806262 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.806242 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r8pxs" Jul 23 00:28:09.822163 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:28:09.822126 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0546d769_8f4f_4c68_be13_64bab60debb1.slice/crio-9a66c44170ed99550c9f0566222c81d0a53c54a8d1d2765f5fd48d1ad851573a WatchSource:0}: Error finding container 9a66c44170ed99550c9f0566222c81d0a53c54a8d1d2765f5fd48d1ad851573a: Status 404 returned error can't find the container with id 9a66c44170ed99550c9f0566222c81d0a53c54a8d1d2765f5fd48d1ad851573a Jul 23 00:28:09.834091 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.834000 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:09.887573 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.887516 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2xg4" event={"ID":"cc4dfee5-f1cd-4494-90ca-c68bfef5f363","Type":"ContainerStarted","Data":"dc47b18ad399df67f00892752340b416ac66f0111d0e0e7cd05e9d2211aa90ee"} Jul 23 00:28:09.889559 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.889344 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2gcnv" Jul 23 00:28:09.889735 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.889713 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8pxs" event={"ID":"0546d769-8f4f-4c68-be13-64bab60debb1","Type":"ContainerStarted","Data":"9a66c44170ed99550c9f0566222c81d0a53c54a8d1d2765f5fd48d1ad851573a"} Jul 23 00:28:09.894622 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.894579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:09.912120 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.912018 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p2xg4" podStartSLOduration=32.86033838 podStartE2EDuration="44.911998777s" totalImagesPullingTime="12.051660397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 00:27:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:09.910385541 +0000 UTC m=+282.406054258" watchObservedRunningTime="2026-07-23 00:28:09.911998777 +0000 UTC m=+282.407667490" Jul 23 00:28:09.934766 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.934736 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6"] Jul 23 00:28:09.936947 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:28:09.936920 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddac8ecd_70f8_498d_a8a9_eb9ff2278241.slice/crio-01241816a2d3aabbe32d92d28112f2fb09ca6e9c4e17352e208dc58f4c852e6e WatchSource:0}: Error finding container 01241816a2d3aabbe32d92d28112f2fb09ca6e9c4e17352e208dc58f4c852e6e: Status 404 returned error can't find the container with id 01241816a2d3aabbe32d92d28112f2fb09ca6e9c4e17352e208dc58f4c852e6e Jul 23 00:28:09.958330 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:09.958305 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-jm5s2"] Jul 23 00:28:09.968622 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:28:09.968598 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066d1d20_c2c9_4a0f_8979_bd68e53154b7.slice/crio-977b792507bf4220575cf08761b853bd3562dd1e2d56edbb1fb95f2aec2e87b0 WatchSource:0}: Error finding container 977b792507bf4220575cf08761b853bd3562dd1e2d56edbb1fb95f2aec2e87b0: Status 404 returned error can't find the container with id 977b792507bf4220575cf08761b853bd3562dd1e2d56edbb1fb95f2aec2e87b0 Jul 23 00:28:10.265493 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:10.265460 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:10.896383 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:10.896340 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" event={"ID":"ddac8ecd-70f8-498d-a8a9-eb9ff2278241","Type":"ContainerStarted","Data":"7ea2b5bc16024bb11608967c4a5438e998f56c08533bc42ac1d31b3f42e35abf"} Jul 23 00:28:10.896801 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:10.896419 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" event={"ID":"ddac8ecd-70f8-498d-a8a9-eb9ff2278241","Type":"ContainerStarted","Data":"c75f85ee1192e64f6f9a3e7082d83c3b002dc583549595a5895044c023b8ef63"} Jul 23 00:28:10.896801 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:10.896449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" event={"ID":"ddac8ecd-70f8-498d-a8a9-eb9ff2278241","Type":"ContainerStarted","Data":"01241816a2d3aabbe32d92d28112f2fb09ca6e9c4e17352e208dc58f4c852e6e"} Jul 23 00:28:10.897745 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:10.897717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" event={"ID":"066d1d20-c2c9-4a0f-8979-bd68e53154b7","Type":"ContainerStarted","Data":"977b792507bf4220575cf08761b853bd3562dd1e2d56edbb1fb95f2aec2e87b0"} Jul 23 00:28:11.764909 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.764874 2564 patch_prober.go:28] interesting pod/image-registry-77bf5fc84-r6bwx container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 23 00:28:11.765069 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.764932 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" podUID="42166b50-1d69-4b6f-a926-0a9f61f76c60" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 00:28:11.781753 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.781724 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:11.852358 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.852324 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6575888f5c-dqsr5"] Jul 23 00:28:11.856586 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.856549 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.857814 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.857724 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:11.860148 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.859943 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 23 00:28:11.860148 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.859960 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 23 00:28:11.860148 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.859974 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 23 00:28:11.860148 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.859960 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 23 00:28:11.860835 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.860696 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9d781equieu2c\"" Jul 23 00:28:11.860835 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.860722 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 23 00:28:11.860835 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.860733 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cmrhh\"" Jul 23 00:28:11.874666 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.874610 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6575888f5c-dqsr5"] Jul 23 00:28:11.880031 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.880011 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-grpc-tls\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.880144 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.880049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht6st\" (UniqueName: \"kubernetes.io/projected/630d7df4-b577-4d44-a4b5-204656fceb1e-kube-api-access-ht6st\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.880144 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.880123 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.880251 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.880173 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.880251 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.880192 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-thanos-querier-tls\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.880251 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.880219 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.880251 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.880245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.880437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.880301 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/630d7df4-b577-4d44-a4b5-204656fceb1e-metrics-client-ca\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.907323 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.907288 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" event={"ID":"ddac8ecd-70f8-498d-a8a9-eb9ff2278241","Type":"ContainerStarted","Data":"eec43fe44d8d5722e6af23f1f044d03965045eedd943a413369f0ec093b367a9"} Jul 23 00:28:11.910449 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.910420 2564 generic.go:356] "Generic (PLEG): container finished" podID="0546d769-8f4f-4c68-be13-64bab60debb1" containerID="8833d980fdb8f6774acd9b08209bbbab906a1bb5626d5438abe21d97f3988b02" exitCode=0 Jul 23 00:28:11.912708 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.911296 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8pxs" event={"ID":"0546d769-8f4f-4c68-be13-64bab60debb1","Type":"ContainerDied","Data":"8833d980fdb8f6774acd9b08209bbbab906a1bb5626d5438abe21d97f3988b02"} Jul 23 00:28:11.917467 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.917437 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" event={"ID":"066d1d20-c2c9-4a0f-8979-bd68e53154b7","Type":"ContainerStarted","Data":"77bdeef7dfaa1aada4c13808048dfd9d9094c2a88ce2be37feb0271d53aec774"} Jul 23 00:28:11.917611 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.917488 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" event={"ID":"066d1d20-c2c9-4a0f-8979-bd68e53154b7","Type":"ContainerStarted","Data":"81ea0d27602fb358ee9077bdb7bca241c272f7656788a955d4b7c093aa0afee8"} Jul 23 00:28:11.917611 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.917508 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" event={"ID":"066d1d20-c2c9-4a0f-8979-bd68e53154b7","Type":"ContainerStarted","Data":"72ce8b32396939bc875cef4ae923be4e2074eb4121cc4557ab48b5e2e05edd68"} Jul 23 00:28:11.932366 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.932316 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-cj7k6" podStartSLOduration=2.617127192 podStartE2EDuration="3.932296945s" totalImagesPullingTime="1.315169753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:11.93172704 +0000 UTC m=+284.427395772" watchObservedRunningTime="2026-07-23 00:28:11.932296945 +0000 UTC m=+284.427965660" Jul 23 00:28:11.969862 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.969809 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-jm5s2" podStartSLOduration=2.55255282 podStartE2EDuration="3.969786428s" totalImagesPullingTime="1.417233608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:11.967287429 +0000 UTC m=+284.462956191" watchObservedRunningTime="2026-07-23 00:28:11.969786428 +0000 UTC m=+284.465455121" Jul 23 00:28:11.981868 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.981372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.981868 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.981573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.981868 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.981636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-thanos-querier-tls\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.981868 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.981664 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.981868 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.981690 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.981868 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.981718 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/630d7df4-b577-4d44-a4b5-204656fceb1e-metrics-client-ca\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.981868 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.981746 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-grpc-tls\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.981868 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.981819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht6st\" (UniqueName: \"kubernetes.io/projected/630d7df4-b577-4d44-a4b5-204656fceb1e-kube-api-access-ht6st\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.983495 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.983473 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/630d7df4-b577-4d44-a4b5-204656fceb1e-metrics-client-ca\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.984429 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.984192 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.985251 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.985224 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.986215 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.986189 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.986835 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.986807 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-grpc-tls\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.986961 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.986940 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-thanos-querier-tls\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.987670 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.987646 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/630d7df4-b577-4d44-a4b5-204656fceb1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:11.991339 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:11.991321 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht6st\" (UniqueName: \"kubernetes.io/projected/630d7df4-b577-4d44-a4b5-204656fceb1e-kube-api-access-ht6st\") pod \"thanos-querier-6575888f5c-dqsr5\" (UID: \"630d7df4-b577-4d44-a4b5-204656fceb1e\") " pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:12.149723 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:12.149684 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:12.168522 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:12.168494 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:12.292670 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:12.292639 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6575888f5c-dqsr5"] Jul 23 00:28:12.295590 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:28:12.295561 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630d7df4_b577_4d44_a4b5_204656fceb1e.slice/crio-b9de91cf39b084f4653cbd7ea98ba5bb3f3d1b1205ad562aaecba1d2514246c4 WatchSource:0}: Error finding container b9de91cf39b084f4653cbd7ea98ba5bb3f3d1b1205ad562aaecba1d2514246c4: Status 404 returned error can't find the container with id b9de91cf39b084f4653cbd7ea98ba5bb3f3d1b1205ad562aaecba1d2514246c4 Jul 23 00:28:12.482980 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:12.482912 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:12.788028 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:12.787951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:28:12.790207 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:12.790189 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cba4baf3-9d99-4bb1-a822-8d80d49c1f9a-metrics-tls\") pod \"dns-default-2gcnv\" (UID: \"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a\") " pod="openshift-dns/dns-default-2gcnv" Jul 23 00:28:12.893585 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:12.893554 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fhdn8\"" Jul 23 00:28:12.900863 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:12.900844 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2gcnv" Jul 23 00:28:12.923384 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:12.923354 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" event={"ID":"630d7df4-b577-4d44-a4b5-204656fceb1e","Type":"ContainerStarted","Data":"b9de91cf39b084f4653cbd7ea98ba5bb3f3d1b1205ad562aaecba1d2514246c4"} Jul 23 00:28:12.925875 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:12.925827 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8pxs" event={"ID":"0546d769-8f4f-4c68-be13-64bab60debb1","Type":"ContainerStarted","Data":"c7582a9b0a1cec25d1d0e4389f7c747b0c5afccbefe5a6f2880fff1ddb53da46"} Jul 23 00:28:12.925992 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:12.925904 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8pxs" event={"ID":"0546d769-8f4f-4c68-be13-64bab60debb1","Type":"ContainerStarted","Data":"82736422e371da01d9c0ea446f93cfa9f18273cbeb1f12b2437df4e0dd792fde"} Jul 23 00:28:12.945555 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:12.945375 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r8pxs" podStartSLOduration=3.875756091 podStartE2EDuration="4.945355906s" totalImagesPullingTime="984.507396ms" totalInitContainerRuntime="85.092419ms" isStatefulPod=true podCreationTimestamp="2026-07-23 00:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:12.94411735 +0000 UTC m=+285.439786062" watchObservedRunningTime="2026-07-23 00:28:12.945355906 +0000 UTC m=+285.441024619" Jul 23 00:28:13.042485 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.042407 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2gcnv"] Jul 23 00:28:13.045712 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:28:13.045679 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba4baf3_9d99_4bb1_a822_8d80d49c1f9a.slice/crio-4cf63a599ea195e70129f7c5214fedc0dc845982069a1955718cd83e3b0f0097 WatchSource:0}: Error finding container 4cf63a599ea195e70129f7c5214fedc0dc845982069a1955718cd83e3b0f0097: Status 404 returned error can't find the container with id 4cf63a599ea195e70129f7c5214fedc0dc845982069a1955718cd83e3b0f0097 Jul 23 00:28:13.157775 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.157742 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-75d4cd5994-995z5"] Jul 23 00:28:13.161380 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.161357 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.164457 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.164203 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 23 00:28:13.164457 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.164222 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gb74p\"" Jul 23 00:28:13.164457 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.164286 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 23 00:28:13.164457 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.164288 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-45t4qv2df2oe7\"" Jul 23 00:28:13.164457 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.164352 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 23 00:28:13.165094 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.165074 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 23 00:28:13.172521 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.172498 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75d4cd5994-995z5"] Jul 23 00:28:13.190573 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.190551 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qck6\" (UniqueName: \"kubernetes.io/projected/1031cf51-37be-47f2-a48c-3c2bd09bb529-kube-api-access-5qck6\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.190679 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.190588 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1031cf51-37be-47f2-a48c-3c2bd09bb529-secret-metrics-server-tls\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.190679 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.190633 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1031cf51-37be-47f2-a48c-3c2bd09bb529-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.190794 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.190712 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1031cf51-37be-47f2-a48c-3c2bd09bb529-secret-metrics-server-client-certs\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.190794 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.190735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1031cf51-37be-47f2-a48c-3c2bd09bb529-metrics-server-audit-profiles\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.190794 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.190751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1031cf51-37be-47f2-a48c-3c2bd09bb529-client-ca-bundle\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.190937 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.190811 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1031cf51-37be-47f2-a48c-3c2bd09bb529-audit-log\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.291798 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.291764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1031cf51-37be-47f2-a48c-3c2bd09bb529-audit-log\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.291923 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.291808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qck6\" (UniqueName: \"kubernetes.io/projected/1031cf51-37be-47f2-a48c-3c2bd09bb529-kube-api-access-5qck6\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.291923 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.291835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1031cf51-37be-47f2-a48c-3c2bd09bb529-secret-metrics-server-tls\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.291923 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.291869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1031cf51-37be-47f2-a48c-3c2bd09bb529-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.291923 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.291890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1031cf51-37be-47f2-a48c-3c2bd09bb529-secret-metrics-server-client-certs\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.291923 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.291907 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1031cf51-37be-47f2-a48c-3c2bd09bb529-metrics-server-audit-profiles\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.292165 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.291929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1031cf51-37be-47f2-a48c-3c2bd09bb529-client-ca-bundle\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.292399 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.292234 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1031cf51-37be-47f2-a48c-3c2bd09bb529-audit-log\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.292739 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.292683 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1031cf51-37be-47f2-a48c-3c2bd09bb529-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.293002 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.292983 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1031cf51-37be-47f2-a48c-3c2bd09bb529-metrics-server-audit-profiles\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.294435 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.294410 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1031cf51-37be-47f2-a48c-3c2bd09bb529-secret-metrics-server-client-certs\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.294516 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.294483 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1031cf51-37be-47f2-a48c-3c2bd09bb529-client-ca-bundle\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.294580 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.294516 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1031cf51-37be-47f2-a48c-3c2bd09bb529-secret-metrics-server-tls\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.310518 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.310499 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qck6\" (UniqueName: \"kubernetes.io/projected/1031cf51-37be-47f2-a48c-3c2bd09bb529-kube-api-access-5qck6\") pod \"metrics-server-75d4cd5994-995z5\" (UID: \"1031cf51-37be-47f2-a48c-3c2bd09bb529\") " pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.474008 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.473969 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:13.626879 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.626795 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-75d4cd5994-995z5"] Jul 23 00:28:13.627718 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.627523 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:13.631108 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:28:13.631081 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1031cf51_37be_47f2_a48c_3c2bd09bb529.slice/crio-2c27dff9b99881fcd0f312c74edfbac1ce5667a1485c62ca54f787c63c9eb41c WatchSource:0}: Error finding container 2c27dff9b99881fcd0f312c74edfbac1ce5667a1485c62ca54f787c63c9eb41c: Status 404 returned error can't find the container with id 2c27dff9b99881fcd0f312c74edfbac1ce5667a1485c62ca54f787c63c9eb41c Jul 23 00:28:13.934274 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.934217 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" event={"ID":"1031cf51-37be-47f2-a48c-3c2bd09bb529","Type":"ContainerStarted","Data":"2c27dff9b99881fcd0f312c74edfbac1ce5667a1485c62ca54f787c63c9eb41c"} Jul 23 00:28:13.935896 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.935864 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2gcnv" event={"ID":"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a","Type":"ContainerStarted","Data":"4cf63a599ea195e70129f7c5214fedc0dc845982069a1955718cd83e3b0f0097"} Jul 23 00:28:13.957846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:13.957794 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:15.017695 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:15.017652 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:15.947900 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:15.947861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" event={"ID":"1031cf51-37be-47f2-a48c-3c2bd09bb529","Type":"ContainerStarted","Data":"121c7a8512d64202322b1d9fb82da45f73994efee5ffefce00aaeec69fddeae6"} Jul 23 00:28:15.950975 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:15.950916 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" event={"ID":"630d7df4-b577-4d44-a4b5-204656fceb1e","Type":"ContainerStarted","Data":"aac51f4b47a3e0495b1fb19f2d3bc7af3840b0163d3641bf2fd5905f5e3fb3f9"} Jul 23 00:28:15.951077 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:15.950980 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" event={"ID":"630d7df4-b577-4d44-a4b5-204656fceb1e","Type":"ContainerStarted","Data":"bb64e0b223a0a0f60e24e172c7652dc33cd8fcb77039a7df9d8f0f17a4710207"} Jul 23 00:28:15.952944 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:15.952920 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2gcnv" event={"ID":"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a","Type":"ContainerStarted","Data":"099e82d046235544deb0834983703fb9846dee88090c04118b68cd3776521340"} Jul 23 00:28:15.953063 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:15.953051 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2gcnv" Jul 23 00:28:15.968751 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:15.968670 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" podStartSLOduration=0.860647867 podStartE2EDuration="2.96865675s" totalImagesPullingTime="2.108008883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:28:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:15.967045529 +0000 UTC m=+288.462714242" watchObservedRunningTime="2026-07-23 00:28:15.96865675 +0000 UTC m=+288.464325464" Jul 23 00:28:15.984991 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:15.984941 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2gcnv" podStartSLOduration=251.294039695 podStartE2EDuration="4m13.9849225s" totalImagesPullingTime="2.690882805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:15.983747859 +0000 UTC m=+288.479416572" watchObservedRunningTime="2026-07-23 00:28:15.9849225 +0000 UTC m=+288.480591210" Jul 23 00:28:16.777758 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:16.777725 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" podUID="42166b50-1d69-4b6f-a926-0a9f61f76c60" containerName="registry" containerID="cri-o://80da79d18ddcc478d608e462fe9fc40c1ca40ff1b20574cb5909ce5b678d520f" gracePeriod=30 Jul 23 00:28:16.958235 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:16.958206 2564 generic.go:356] "Generic (PLEG): container finished" podID="42166b50-1d69-4b6f-a926-0a9f61f76c60" containerID="80da79d18ddcc478d608e462fe9fc40c1ca40ff1b20574cb5909ce5b678d520f" exitCode=0 Jul 23 00:28:16.958345 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:16.958274 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" event={"ID":"42166b50-1d69-4b6f-a926-0a9f61f76c60","Type":"ContainerDied","Data":"80da79d18ddcc478d608e462fe9fc40c1ca40ff1b20574cb5909ce5b678d520f"} Jul 23 00:28:16.960615 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:16.960595 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" event={"ID":"630d7df4-b577-4d44-a4b5-204656fceb1e","Type":"ContainerStarted","Data":"94830055f6150279ab8ecc4960088bce43fa00dbbd5262b8eee2d76293ae508d"} Jul 23 00:28:16.960664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:16.960648 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" event={"ID":"630d7df4-b577-4d44-a4b5-204656fceb1e","Type":"ContainerStarted","Data":"43455ba344dbe0a5c5cfe1a047b28fd882f125b42e344589af0137dfb8282537"} Jul 23 00:28:16.960701 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:16.960673 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" event={"ID":"630d7df4-b577-4d44-a4b5-204656fceb1e","Type":"ContainerStarted","Data":"f0c0771959c707986a75c3af0b71303f55b59faa72a2dd95764d99d316fbdef1"} Jul 23 00:28:16.960701 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:16.960696 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" event={"ID":"630d7df4-b577-4d44-a4b5-204656fceb1e","Type":"ContainerStarted","Data":"f012a27e5114a6c6c0599de82c6b08b13e0da473e852c8d51ef2c6ed4397a2df"} Jul 23 00:28:16.961557 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:16.961400 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:16.968996 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:16.968885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2gcnv" event={"ID":"cba4baf3-9d99-4bb1-a822-8d80d49c1f9a","Type":"ContainerStarted","Data":"15d39087fd2cddd04b3f783cfc76266090fdd63440657638980cd0f5fda25ae6"} Jul 23 00:28:16.990980 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:16.990923 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" podStartSLOduration=1.786459759 podStartE2EDuration="5.99090552s" totalImagesPullingTime="4.204445761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:28:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:16.988092252 +0000 UTC m=+289.483760964" watchObservedRunningTime="2026-07-23 00:28:16.99090552 +0000 UTC m=+289.486574233" Jul 23 00:28:17.019783 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.019763 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:28:17.127327 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.127298 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42166b50-1d69-4b6f-a926-0a9f61f76c60-trusted-ca\") pod \"42166b50-1d69-4b6f-a926-0a9f61f76c60\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " Jul 23 00:28:17.127327 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.127330 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls\") pod \"42166b50-1d69-4b6f-a926-0a9f61f76c60\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " Jul 23 00:28:17.127486 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.127352 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42166b50-1d69-4b6f-a926-0a9f61f76c60-ca-trust-extracted\") pod \"42166b50-1d69-4b6f-a926-0a9f61f76c60\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " Jul 23 00:28:17.127486 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.127379 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-certificates\") pod \"42166b50-1d69-4b6f-a926-0a9f61f76c60\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " Jul 23 00:28:17.127593 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.127515 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42166b50-1d69-4b6f-a926-0a9f61f76c60-image-registry-private-configuration\") pod \"42166b50-1d69-4b6f-a926-0a9f61f76c60\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " Jul 23 00:28:17.127637 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.127605 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp95n\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-kube-api-access-gp95n\") pod \"42166b50-1d69-4b6f-a926-0a9f61f76c60\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " Jul 23 00:28:17.127687 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.127639 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-bound-sa-token\") pod \"42166b50-1d69-4b6f-a926-0a9f61f76c60\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " Jul 23 00:28:17.127745 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.127719 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42166b50-1d69-4b6f-a926-0a9f61f76c60-installation-pull-secrets\") pod \"42166b50-1d69-4b6f-a926-0a9f61f76c60\" (UID: \"42166b50-1d69-4b6f-a926-0a9f61f76c60\") " Jul 23 00:28:17.128019 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.127890 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42166b50-1d69-4b6f-a926-0a9f61f76c60-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "42166b50-1d69-4b6f-a926-0a9f61f76c60" (UID: "42166b50-1d69-4b6f-a926-0a9f61f76c60"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:28:17.128019 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.127980 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "42166b50-1d69-4b6f-a926-0a9f61f76c60" (UID: "42166b50-1d69-4b6f-a926-0a9f61f76c60"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:28:17.129997 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.129967 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42166b50-1d69-4b6f-a926-0a9f61f76c60-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "42166b50-1d69-4b6f-a926-0a9f61f76c60" (UID: "42166b50-1d69-4b6f-a926-0a9f61f76c60"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:28:17.130124 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.130090 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42166b50-1d69-4b6f-a926-0a9f61f76c60-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "42166b50-1d69-4b6f-a926-0a9f61f76c60" (UID: "42166b50-1d69-4b6f-a926-0a9f61f76c60"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:28:17.130235 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.130177 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "42166b50-1d69-4b6f-a926-0a9f61f76c60" (UID: "42166b50-1d69-4b6f-a926-0a9f61f76c60"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:28:17.130235 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.130183 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-kube-api-access-gp95n" (OuterVolumeSpecName: "kube-api-access-gp95n") pod "42166b50-1d69-4b6f-a926-0a9f61f76c60" (UID: "42166b50-1d69-4b6f-a926-0a9f61f76c60"). InnerVolumeSpecName "kube-api-access-gp95n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:28:17.130235 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.130213 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "42166b50-1d69-4b6f-a926-0a9f61f76c60" (UID: "42166b50-1d69-4b6f-a926-0a9f61f76c60"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:28:17.135420 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.135398 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42166b50-1d69-4b6f-a926-0a9f61f76c60-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "42166b50-1d69-4b6f-a926-0a9f61f76c60" (UID: "42166b50-1d69-4b6f-a926-0a9f61f76c60"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:28:17.229099 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.229042 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42166b50-1d69-4b6f-a926-0a9f61f76c60-installation-pull-secrets\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:28:17.229099 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.229065 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42166b50-1d69-4b6f-a926-0a9f61f76c60-trusted-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:28:17.229099 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.229075 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-tls\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:28:17.229099 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.229084 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42166b50-1d69-4b6f-a926-0a9f61f76c60-ca-trust-extracted\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:28:17.229099 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.229092 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42166b50-1d69-4b6f-a926-0a9f61f76c60-registry-certificates\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:28:17.229099 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.229101 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42166b50-1d69-4b6f-a926-0a9f61f76c60-image-registry-private-configuration\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:28:17.229339 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.229110 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp95n\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-kube-api-access-gp95n\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:28:17.229339 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.229119 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42166b50-1d69-4b6f-a926-0a9f61f76c60-bound-sa-token\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:28:17.526605 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.526508 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:17.627070 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.627032 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:17.680648 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.680610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:17.973496 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.973465 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" Jul 23 00:28:17.973969 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.973454 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77bf5fc84-r6bwx" event={"ID":"42166b50-1d69-4b6f-a926-0a9f61f76c60","Type":"ContainerDied","Data":"17f213c1b8b4ed063f60982686a1e0b88d6fef557e0c6732eed9199772013355"} Jul 23 00:28:17.973969 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.973594 2564 scope.go:117] "RemoveContainer" containerID="80da79d18ddcc478d608e462fe9fc40c1ca40ff1b20574cb5909ce5b678d520f" Jul 23 00:28:17.994515 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.994485 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77bf5fc84-r6bwx"] Jul 23 00:28:17.998570 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:17.998550 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-77bf5fc84-r6bwx"] Jul 23 00:28:18.071238 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.071216 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42166b50-1d69-4b6f-a926-0a9f61f76c60" path="/var/lib/kubelet/pods/42166b50-1d69-4b6f-a926-0a9f61f76c60/volumes" Jul 23 00:28:18.739224 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.739192 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-h2wtb"] Jul 23 00:28:18.739517 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.739504 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42166b50-1d69-4b6f-a926-0a9f61f76c60" containerName="registry" Jul 23 00:28:18.739585 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.739519 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="42166b50-1d69-4b6f-a926-0a9f61f76c60" containerName="registry" Jul 23 00:28:18.739619 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.739608 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="42166b50-1d69-4b6f-a926-0a9f61f76c60" containerName="registry" Jul 23 00:28:18.742814 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.742788 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-h2wtb" Jul 23 00:28:18.745348 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.745324 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 00:28:18.745469 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.745331 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-vc79j\"" Jul 23 00:28:18.745469 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.745373 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 00:28:18.752125 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.752096 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:18.756306 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.756286 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-h2wtb"] Jul 23 00:28:18.840893 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.840867 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqk9f\" (UniqueName: \"kubernetes.io/projected/908cfd03-f8f2-424a-9802-76218e7d5128-kube-api-access-lqk9f\") pod \"downloads-5bc67dcd46-h2wtb\" (UID: \"908cfd03-f8f2-424a-9802-76218e7d5128\") " pod="openshift-console/downloads-5bc67dcd46-h2wtb" Jul 23 00:28:18.849696 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.849671 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:18.942013 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.941986 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqk9f\" (UniqueName: \"kubernetes.io/projected/908cfd03-f8f2-424a-9802-76218e7d5128-kube-api-access-lqk9f\") pod \"downloads-5bc67dcd46-h2wtb\" (UID: \"908cfd03-f8f2-424a-9802-76218e7d5128\") " pod="openshift-console/downloads-5bc67dcd46-h2wtb" Jul 23 00:28:18.950205 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.950175 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqk9f\" (UniqueName: \"kubernetes.io/projected/908cfd03-f8f2-424a-9802-76218e7d5128-kube-api-access-lqk9f\") pod \"downloads-5bc67dcd46-h2wtb\" (UID: \"908cfd03-f8f2-424a-9802-76218e7d5128\") " pod="openshift-console/downloads-5bc67dcd46-h2wtb" Jul 23 00:28:18.994187 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:18.994111 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:19.051076 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:19.051055 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:19.051936 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:19.051919 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-h2wtb" Jul 23 00:28:19.180489 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:19.180467 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-h2wtb"] Jul 23 00:28:19.182827 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:28:19.182798 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908cfd03_f8f2_424a_9802_76218e7d5128.slice/crio-4287865723f00cef4fa7b6367295630b29b24fc7e8dd1582100cf39d3318827c WatchSource:0}: Error finding container 4287865723f00cef4fa7b6367295630b29b24fc7e8dd1582100cf39d3318827c: Status 404 returned error can't find the container with id 4287865723f00cef4fa7b6367295630b29b24fc7e8dd1582100cf39d3318827c Jul 23 00:28:19.221056 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:19.221030 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:19.589664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:19.589631 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:19.690656 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:19.690627 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:19.984521 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:19.984478 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-h2wtb" event={"ID":"908cfd03-f8f2-424a-9802-76218e7d5128","Type":"ContainerStarted","Data":"4287865723f00cef4fa7b6367295630b29b24fc7e8dd1582100cf39d3318827c"} Jul 23 00:28:20.211502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:20.211451 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:20.267041 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:20.266959 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:20.398087 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:20.398054 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:20.798416 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:20.798381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:21.343409 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:21.343368 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:23.308744 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:23.308711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:23.408958 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:23.408922 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:23.627389 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:23.627272 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:23.680985 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:23.680923 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:23.804083 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:23.804042 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:23.861671 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:23.861640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:23.985888 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:23.985859 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6575888f5c-dqsr5" Jul 23 00:28:25.098307 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:25.098269 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:25.197871 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:25.197826 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:25.745248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:25.745212 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:25.846775 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:25.846736 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:26.976299 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:26.976265 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2gcnv" Jul 23 00:28:27.823094 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.823060 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f96b549c-w57wt"] Jul 23 00:28:27.828512 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.828484 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:27.831128 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.831105 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 00:28:27.831291 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.831274 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 00:28:27.831364 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.831339 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 00:28:27.832357 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.832337 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 00:28:27.832431 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.832360 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 00:28:27.832431 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.832402 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wx88k\"" Jul 23 00:28:27.838885 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.838835 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f96b549c-w57wt"] Jul 23 00:28:27.919189 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.919159 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a442c73-2530-44fc-a390-3d059bc78a0b-console-oauth-config\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:27.919402 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.919383 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:28:27.919486 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.919441 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a442c73-2530-44fc-a390-3d059bc78a0b-console-serving-cert\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:27.919566 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.919486 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-service-ca\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:27.919566 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.919520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-oauth-serving-cert\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:27.919566 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.919554 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:28:27.919740 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.919604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdlpd\" (UniqueName: \"kubernetes.io/projected/4a442c73-2530-44fc-a390-3d059bc78a0b-kube-api-access-pdlpd\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:27.919740 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.919633 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-console-config\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:27.922658 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:27.922642 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 00:28:28.020582 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.020447 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a442c73-2530-44fc-a390-3d059bc78a0b-console-serving-cert\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:28.020582 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.020479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-service-ca\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:28.020582 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.020502 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-oauth-serving-cert\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:28.020582 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.020553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdlpd\" (UniqueName: \"kubernetes.io/projected/4a442c73-2530-44fc-a390-3d059bc78a0b-kube-api-access-pdlpd\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:28.020582 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.020574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-console-config\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:28.029673 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.020595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a442c73-2530-44fc-a390-3d059bc78a0b-console-oauth-config\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:28.029673 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.023097 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 00:28:28.029673 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.023207 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 00:28:28.029673 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.023236 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 00:28:28.029673 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.023261 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 00:28:28.029673 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.023737 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 00:28:28.029949 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.029927 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdlpd\" (UniqueName: \"kubernetes.io/projected/4a442c73-2530-44fc-a390-3d059bc78a0b-kube-api-access-pdlpd\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:28.031717 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.031673 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-service-ca\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:28.031833 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.031784 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-oauth-serving-cert\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:28.031833 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.031801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-console-config\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:28.033857 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.033833 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a442c73-2530-44fc-a390-3d059bc78a0b-console-oauth-config\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:28.033970 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.033928 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a442c73-2530-44fc-a390-3d059bc78a0b-console-serving-cert\") pod \"console-6f96b549c-w57wt\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:28.140448 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.140420 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wx88k\"" Jul 23 00:28:28.148585 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.148564 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:28.310124 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:28.310050 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f96b549c-w57wt"] Jul 23 00:28:28.312711 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:28:28.312680 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a442c73_2530_44fc_a390_3d059bc78a0b.slice/crio-36348ad5f8f4f1ca4402f839cea32f8da31085894965033fa9fa6df9cded9314 WatchSource:0}: Error finding container 36348ad5f8f4f1ca4402f839cea32f8da31085894965033fa9fa6df9cded9314: Status 404 returned error can't find the container with id 36348ad5f8f4f1ca4402f839cea32f8da31085894965033fa9fa6df9cded9314 Jul 23 00:28:29.024621 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:29.024579 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f96b549c-w57wt" event={"ID":"4a442c73-2530-44fc-a390-3d059bc78a0b","Type":"ContainerStarted","Data":"36348ad5f8f4f1ca4402f839cea32f8da31085894965033fa9fa6df9cded9314"} Jul 23 00:28:29.841175 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:29.841141 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f4bd66545-szh5g"] Jul 23 00:28:29.869426 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:29.869392 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:29.902600 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:29.902566 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f4bd66545-szh5g"] Jul 23 00:28:29.902765 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:29.902695 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:29.913636 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:29.913287 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 23 00:28:29.949631 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:29.949597 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:30.039342 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.039313 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-console-config\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.039784 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.039392 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/957f3964-09d1-44f9-8a13-2070edd225ca-console-oauth-config\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.039784 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.039424 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vth4k\" (UniqueName: \"kubernetes.io/projected/957f3964-09d1-44f9-8a13-2070edd225ca-kube-api-access-vth4k\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.039784 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.039460 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/957f3964-09d1-44f9-8a13-2070edd225ca-console-serving-cert\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.039784 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.039496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-service-ca\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.039784 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.039547 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-trusted-ca-bundle\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.039784 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.039587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-oauth-serving-cert\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.140717 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.140675 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/957f3964-09d1-44f9-8a13-2070edd225ca-console-oauth-config\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.140717 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.140725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vth4k\" (UniqueName: \"kubernetes.io/projected/957f3964-09d1-44f9-8a13-2070edd225ca-kube-api-access-vth4k\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.140952 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.140767 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/957f3964-09d1-44f9-8a13-2070edd225ca-console-serving-cert\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.140952 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.140797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-service-ca\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.140952 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.140824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-trusted-ca-bundle\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.140952 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.140860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-oauth-serving-cert\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.140952 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.140902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-console-config\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.142849 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.142662 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-trusted-ca-bundle\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.142849 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.142789 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-console-config\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.143048 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.142883 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-oauth-serving-cert\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.143048 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.142919 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-service-ca\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.143654 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.143633 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/957f3964-09d1-44f9-8a13-2070edd225ca-console-oauth-config\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.143875 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.143850 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/957f3964-09d1-44f9-8a13-2070edd225ca-console-serving-cert\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.159347 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.159321 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vth4k\" (UniqueName: \"kubernetes.io/projected/957f3964-09d1-44f9-8a13-2070edd225ca-kube-api-access-vth4k\") pod \"console-6f4bd66545-szh5g\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.216551 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.216498 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:30.445550 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:30.445440 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:32.447485 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:32.447435 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:32.506870 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:32.506824 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:33.474287 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:33.474247 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:33.474785 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:33.474319 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:34.401225 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:34.401183 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:34.500678 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:34.500638 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:34.896605 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:34.896573 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:38.294848 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:38.294825 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f4bd66545-szh5g"] Jul 23 00:28:38.297741 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:28:38.297718 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957f3964_09d1_44f9_8a13_2070edd225ca.slice/crio-50e025d8bba6aeb7552c3349074c8d8500831da311317c26cd13b9c1bb7ce8e6 WatchSource:0}: Error finding container 50e025d8bba6aeb7552c3349074c8d8500831da311317c26cd13b9c1bb7ce8e6: Status 404 returned error can't find the container with id 50e025d8bba6aeb7552c3349074c8d8500831da311317c26cd13b9c1bb7ce8e6 Jul 23 00:28:39.072219 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:39.072179 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4bd66545-szh5g" event={"ID":"957f3964-09d1-44f9-8a13-2070edd225ca","Type":"ContainerStarted","Data":"a59cc2f21f4ef353170fbcd4ed48aa6ee38d11fe1eec019f18862f6037b1b742"} Jul 23 00:28:39.072420 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:39.072237 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4bd66545-szh5g" event={"ID":"957f3964-09d1-44f9-8a13-2070edd225ca","Type":"ContainerStarted","Data":"50e025d8bba6aeb7552c3349074c8d8500831da311317c26cd13b9c1bb7ce8e6"} Jul 23 00:28:39.074072 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:39.074032 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-h2wtb" event={"ID":"908cfd03-f8f2-424a-9802-76218e7d5128","Type":"ContainerStarted","Data":"c4700e72c54875861e0478cabed767e97735808fcb707df9ded8de83657ef7e0"} Jul 23 00:28:39.074203 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:39.074094 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-h2wtb" Jul 23 00:28:39.076220 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:39.076131 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f96b549c-w57wt" event={"ID":"4a442c73-2530-44fc-a390-3d059bc78a0b","Type":"ContainerStarted","Data":"3f6932650737d0944f8e5078f4ec62915193d5be4b2e0648b3424fb1e15e7677"} Jul 23 00:28:39.088139 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:39.088108 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-h2wtb" Jul 23 00:28:39.091253 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:39.091198 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f4bd66545-szh5g" podStartSLOduration=10.09117874 podStartE2EDuration="10.09117874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:39.089171683 +0000 UTC m=+311.584840418" watchObservedRunningTime="2026-07-23 00:28:39.09117874 +0000 UTC m=+311.586847495" Jul 23 00:28:39.109243 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:39.109187 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f96b549c-w57wt" podStartSLOduration=2.251226939 podStartE2EDuration="12.109167959s" totalImagesPullingTime="9.85794102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:39.108085699 +0000 UTC m=+311.603754412" watchObservedRunningTime="2026-07-23 00:28:39.109167959 +0000 UTC m=+311.604836737" Jul 23 00:28:39.130728 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:39.130652 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-h2wtb" podStartSLOduration=2.123272401 podStartE2EDuration="21.130630243s" totalImagesPullingTime="19.007357842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:28:39.129340951 +0000 UTC m=+311.625009698" watchObservedRunningTime="2026-07-23 00:28:39.130630243 +0000 UTC m=+311.626298955" Jul 23 00:28:40.217234 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:40.217192 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:40.217727 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:40.217271 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:40.222949 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:40.222917 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:41.090651 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:41.090617 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:28:41.168744 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:41.168711 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f96b549c-w57wt"] Jul 23 00:28:41.474728 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:41.474685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:41.561239 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:41.561206 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:42.497166 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:42.497132 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:42.597499 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:42.597466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:43.894318 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:43.894271 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:43.943827 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:43.943790 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:48.149740 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:48.149699 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:28:49.416235 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:49.416200 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:49.517178 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:49.517145 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:28:51.128990 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:51.128960 2564 generic.go:356] "Generic (PLEG): container finished" podID="436060b9-c207-4a56-a70e-65e1a51c3b17" containerID="51bb1ed082e06a63af50eab7800d41985b9908bf080be5e7dc2db31a049ea5ba" exitCode=0 Jul 23 00:28:51.129342 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:51.129005 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-wjhdx" event={"ID":"436060b9-c207-4a56-a70e-65e1a51c3b17","Type":"ContainerDied","Data":"51bb1ed082e06a63af50eab7800d41985b9908bf080be5e7dc2db31a049ea5ba"} Jul 23 00:28:51.129342 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:51.129267 2564 scope.go:117] "RemoveContainer" containerID="51bb1ed082e06a63af50eab7800d41985b9908bf080be5e7dc2db31a049ea5ba" Jul 23 00:28:52.134158 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:52.134123 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-wjhdx" event={"ID":"436060b9-c207-4a56-a70e-65e1a51c3b17","Type":"ContainerStarted","Data":"0b33c0d9450a953ccd2701218b7bc7dba0ea52f998fbbff438d1c0078373b9e6"} Jul 23 00:28:53.479852 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:53.479819 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:53.483829 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:53.483805 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-75d4cd5994-995z5" Jul 23 00:28:57.154685 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:57.154649 2564 generic.go:356] "Generic (PLEG): container finished" podID="bf5354e7-1133-4900-93f6-dbf7fddeed5b" containerID="fc42627d3ac0e663bc0db2411bfdff975118ec727899870a4e978d4111b941b0" exitCode=0 Jul 23 00:28:57.155071 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:57.154720 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" event={"ID":"bf5354e7-1133-4900-93f6-dbf7fddeed5b","Type":"ContainerDied","Data":"fc42627d3ac0e663bc0db2411bfdff975118ec727899870a4e978d4111b941b0"} Jul 23 00:28:57.155071 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:57.155031 2564 scope.go:117] "RemoveContainer" containerID="fc42627d3ac0e663bc0db2411bfdff975118ec727899870a4e978d4111b941b0" Jul 23 00:28:58.160416 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:28:58.160378 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-x8wz6" event={"ID":"bf5354e7-1133-4900-93f6-dbf7fddeed5b","Type":"ContainerStarted","Data":"775b28332bbf7ed9b6b6bf30be50a6884b595c3338844878d9f3d61aaae56d84"} Jul 23 00:29:04.416245 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:04.416200 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:04.471076 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:04.471034 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:06.193229 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.193169 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f96b549c-w57wt" podUID="4a442c73-2530-44fc-a390-3d059bc78a0b" containerName="console" containerID="cri-o://3f6932650737d0944f8e5078f4ec62915193d5be4b2e0648b3424fb1e15e7677" gracePeriod=15 Jul 23 00:29:06.480986 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.480949 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:29:06.567748 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.567703 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdlpd\" (UniqueName: \"kubernetes.io/projected/4a442c73-2530-44fc-a390-3d059bc78a0b-kube-api-access-pdlpd\") pod \"4a442c73-2530-44fc-a390-3d059bc78a0b\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " Jul 23 00:29:06.567953 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.567764 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-oauth-serving-cert\") pod \"4a442c73-2530-44fc-a390-3d059bc78a0b\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " Jul 23 00:29:06.567953 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.567796 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a442c73-2530-44fc-a390-3d059bc78a0b-console-serving-cert\") pod \"4a442c73-2530-44fc-a390-3d059bc78a0b\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " Jul 23 00:29:06.567953 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.567820 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-console-config\") pod \"4a442c73-2530-44fc-a390-3d059bc78a0b\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " Jul 23 00:29:06.567953 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.567846 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-service-ca\") pod \"4a442c73-2530-44fc-a390-3d059bc78a0b\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " Jul 23 00:29:06.567953 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.567886 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a442c73-2530-44fc-a390-3d059bc78a0b-console-oauth-config\") pod \"4a442c73-2530-44fc-a390-3d059bc78a0b\" (UID: \"4a442c73-2530-44fc-a390-3d059bc78a0b\") " Jul 23 00:29:06.568263 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.568221 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-service-ca" (OuterVolumeSpecName: "service-ca") pod "4a442c73-2530-44fc-a390-3d059bc78a0b" (UID: "4a442c73-2530-44fc-a390-3d059bc78a0b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:29:06.568325 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.568260 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4a442c73-2530-44fc-a390-3d059bc78a0b" (UID: "4a442c73-2530-44fc-a390-3d059bc78a0b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:29:06.568325 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.568264 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-console-config" (OuterVolumeSpecName: "console-config") pod "4a442c73-2530-44fc-a390-3d059bc78a0b" (UID: "4a442c73-2530-44fc-a390-3d059bc78a0b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:29:06.570274 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.570229 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a442c73-2530-44fc-a390-3d059bc78a0b-kube-api-access-pdlpd" (OuterVolumeSpecName: "kube-api-access-pdlpd") pod "4a442c73-2530-44fc-a390-3d059bc78a0b" (UID: "4a442c73-2530-44fc-a390-3d059bc78a0b"). InnerVolumeSpecName "kube-api-access-pdlpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:29:06.570274 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.570234 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a442c73-2530-44fc-a390-3d059bc78a0b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4a442c73-2530-44fc-a390-3d059bc78a0b" (UID: "4a442c73-2530-44fc-a390-3d059bc78a0b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:29:06.570477 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.570310 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a442c73-2530-44fc-a390-3d059bc78a0b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4a442c73-2530-44fc-a390-3d059bc78a0b" (UID: "4a442c73-2530-44fc-a390-3d059bc78a0b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:29:06.669043 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.669007 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdlpd\" (UniqueName: \"kubernetes.io/projected/4a442c73-2530-44fc-a390-3d059bc78a0b-kube-api-access-pdlpd\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:29:06.669043 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.669036 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-oauth-serving-cert\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:29:06.669043 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.669046 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a442c73-2530-44fc-a390-3d059bc78a0b-console-serving-cert\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:29:06.669043 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.669056 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-console-config\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:29:06.669306 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.669065 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a442c73-2530-44fc-a390-3d059bc78a0b-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:29:06.669306 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:06.669072 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a442c73-2530-44fc-a390-3d059bc78a0b-console-oauth-config\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:29:07.203182 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:07.203131 2564 generic.go:356] "Generic (PLEG): container finished" podID="4a442c73-2530-44fc-a390-3d059bc78a0b" containerID="3f6932650737d0944f8e5078f4ec62915193d5be4b2e0648b3424fb1e15e7677" exitCode=0 Jul 23 00:29:07.203182 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:07.203186 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f96b549c-w57wt" Jul 23 00:29:07.203800 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:07.203200 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f96b549c-w57wt" event={"ID":"4a442c73-2530-44fc-a390-3d059bc78a0b","Type":"ContainerDied","Data":"3f6932650737d0944f8e5078f4ec62915193d5be4b2e0648b3424fb1e15e7677"} Jul 23 00:29:07.203800 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:07.203276 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f96b549c-w57wt" event={"ID":"4a442c73-2530-44fc-a390-3d059bc78a0b","Type":"ContainerDied","Data":"36348ad5f8f4f1ca4402f839cea32f8da31085894965033fa9fa6df9cded9314"} Jul 23 00:29:07.203800 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:07.203309 2564 scope.go:117] "RemoveContainer" containerID="3f6932650737d0944f8e5078f4ec62915193d5be4b2e0648b3424fb1e15e7677" Jul 23 00:29:07.213427 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:07.213404 2564 scope.go:117] "RemoveContainer" containerID="3f6932650737d0944f8e5078f4ec62915193d5be4b2e0648b3424fb1e15e7677" Jul 23 00:29:07.213777 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:29:07.213755 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6932650737d0944f8e5078f4ec62915193d5be4b2e0648b3424fb1e15e7677\": container with ID starting with 3f6932650737d0944f8e5078f4ec62915193d5be4b2e0648b3424fb1e15e7677 not found: ID does not exist" containerID="3f6932650737d0944f8e5078f4ec62915193d5be4b2e0648b3424fb1e15e7677" Jul 23 00:29:07.213877 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:07.213791 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6932650737d0944f8e5078f4ec62915193d5be4b2e0648b3424fb1e15e7677"} err="failed to get container status \"3f6932650737d0944f8e5078f4ec62915193d5be4b2e0648b3424fb1e15e7677\": rpc error: code = NotFound desc = could not find container \"3f6932650737d0944f8e5078f4ec62915193d5be4b2e0648b3424fb1e15e7677\": container with ID starting with 3f6932650737d0944f8e5078f4ec62915193d5be4b2e0648b3424fb1e15e7677 not found: ID does not exist" Jul 23 00:29:07.225392 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:07.225363 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f96b549c-w57wt"] Jul 23 00:29:07.227402 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:07.227379 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f96b549c-w57wt"] Jul 23 00:29:08.071996 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:08.071954 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a442c73-2530-44fc-a390-3d059bc78a0b" path="/var/lib/kubelet/pods/4a442c73-2530-44fc-a390-3d059bc78a0b/volumes" Jul 23 00:29:28.008754 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:28.008705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:28.252306 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:28.252208 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:30.988353 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:30.988310 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 00:29:30.988839 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:30.988664 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a442c73-2530-44fc-a390-3d059bc78a0b" containerName="console" Jul 23 00:29:30.988839 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:30.988677 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a442c73-2530-44fc-a390-3d059bc78a0b" containerName="console" Jul 23 00:29:30.988839 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:30.988743 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a442c73-2530-44fc-a390-3d059bc78a0b" containerName="console" Jul 23 00:29:30.992273 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:30.992245 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:30.995208 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:30.995176 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 23 00:29:30.995377 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:30.995176 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 23 00:29:30.995377 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:30.995221 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 23 00:29:30.995606 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:30.995583 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8ckdn\"" Jul 23 00:29:30.995722 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:30.995589 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 23 00:29:30.995722 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:30.995587 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 23 00:29:30.995722 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:30.995691 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 23 00:29:30.995886 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:30.995592 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 23 00:29:30.996009 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:30.995992 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 23 00:29:31.000395 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.000374 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 23 00:29:31.007007 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.006972 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 00:29:31.074893 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.074847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d38addb7-debd-46e7-b185-77d46cd891ec-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.075170 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.074910 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d38addb7-debd-46e7-b185-77d46cd891ec-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.075170 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.074952 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.075170 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.074979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d38addb7-debd-46e7-b185-77d46cd891ec-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.075170 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.075004 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br4v7\" (UniqueName: \"kubernetes.io/projected/d38addb7-debd-46e7-b185-77d46cd891ec-kube-api-access-br4v7\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.075170 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.075031 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-web-config\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.075170 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.075136 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.075462 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.075180 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-config-volume\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.075462 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.075208 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.075462 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.075238 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.075462 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.075269 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d38addb7-debd-46e7-b185-77d46cd891ec-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.075462 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.075319 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d38addb7-debd-46e7-b185-77d46cd891ec-config-out\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.075462 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.075397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.175983 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.175942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.175983 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.175986 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-config-volume\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.176270 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.176010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.176270 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.176112 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.176270 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.176157 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d38addb7-debd-46e7-b185-77d46cd891ec-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.176270 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.176207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d38addb7-debd-46e7-b185-77d46cd891ec-config-out\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.176270 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.176251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.176505 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.176285 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d38addb7-debd-46e7-b185-77d46cd891ec-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.176505 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.176316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d38addb7-debd-46e7-b185-77d46cd891ec-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.176505 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.176354 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.176505 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.176382 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d38addb7-debd-46e7-b185-77d46cd891ec-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.176505 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.176405 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br4v7\" (UniqueName: \"kubernetes.io/projected/d38addb7-debd-46e7-b185-77d46cd891ec-kube-api-access-br4v7\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.176505 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.176432 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-web-config\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.176851 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.176589 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d38addb7-debd-46e7-b185-77d46cd891ec-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.177502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.177473 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d38addb7-debd-46e7-b185-77d46cd891ec-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.178747 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.178718 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d38addb7-debd-46e7-b185-77d46cd891ec-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.179343 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.179315 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-web-config\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.179755 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.179728 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d38addb7-debd-46e7-b185-77d46cd891ec-config-out\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.179886 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.179845 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.179886 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.179855 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.180021 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.179988 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-config-volume\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.180144 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.180004 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.180221 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.180153 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.180959 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.180937 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d38addb7-debd-46e7-b185-77d46cd891ec-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.181368 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.181350 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d38addb7-debd-46e7-b185-77d46cd891ec-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.187659 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.187626 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br4v7\" (UniqueName: \"kubernetes.io/projected/d38addb7-debd-46e7-b185-77d46cd891ec-kube-api-access-br4v7\") pod \"alertmanager-main-0\" (UID: \"d38addb7-debd-46e7-b185-77d46cd891ec\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.302812 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.302707 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 00:29:31.451297 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.451268 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 00:29:31.452908 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:29:31.452878 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38addb7_debd_46e7_b185_77d46cd891ec.slice/crio-05300af2ab255970fa1f85f3457b56b09e1d7d8f256d55e8bd37213a1d338112 WatchSource:0}: Error finding container 05300af2ab255970fa1f85f3457b56b09e1d7d8f256d55e8bd37213a1d338112: Status 404 returned error can't find the container with id 05300af2ab255970fa1f85f3457b56b09e1d7d8f256d55e8bd37213a1d338112 Jul 23 00:29:31.455062 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:31.455046 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:29:32.163089 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:32.163050 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:32.219156 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:32.219101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:32.302731 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:32.302697 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d38addb7-debd-46e7-b185-77d46cd891ec","Type":"ContainerStarted","Data":"05300af2ab255970fa1f85f3457b56b09e1d7d8f256d55e8bd37213a1d338112"} Jul 23 00:29:33.078676 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.078638 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj"] Jul 23 00:29:33.081161 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.081144 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.083483 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.083459 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 23 00:29:33.083795 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.083769 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 23 00:29:33.083909 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.083802 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-b94t8\"" Jul 23 00:29:33.083909 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.083853 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 23 00:29:33.084003 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.083950 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 23 00:29:33.084060 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.084006 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 23 00:29:33.089129 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.089104 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 23 00:29:33.097075 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.097049 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj"] Jul 23 00:29:33.196317 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.196272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ca25d7bb-3038-4662-9616-55e6dcb34f49-telemeter-client-tls\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.196317 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.196317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ca25d7bb-3038-4662-9616-55e6dcb34f49-federate-client-tls\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.196795 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.196341 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca25d7bb-3038-4662-9616-55e6dcb34f49-metrics-client-ca\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.196795 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.196472 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca25d7bb-3038-4662-9616-55e6dcb34f49-serving-certs-ca-bundle\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.196795 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.196522 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca25d7bb-3038-4662-9616-55e6dcb34f49-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.196795 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.196580 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ca25d7bb-3038-4662-9616-55e6dcb34f49-secret-telemeter-client\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.196795 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.196605 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn9bp\" (UniqueName: \"kubernetes.io/projected/ca25d7bb-3038-4662-9616-55e6dcb34f49-kube-api-access-tn9bp\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.196795 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.196645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca25d7bb-3038-4662-9616-55e6dcb34f49-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.297822 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.297779 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ca25d7bb-3038-4662-9616-55e6dcb34f49-telemeter-client-tls\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.297973 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.297829 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ca25d7bb-3038-4662-9616-55e6dcb34f49-federate-client-tls\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.297973 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.297849 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca25d7bb-3038-4662-9616-55e6dcb34f49-metrics-client-ca\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.297973 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.297880 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca25d7bb-3038-4662-9616-55e6dcb34f49-serving-certs-ca-bundle\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.298120 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.298004 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca25d7bb-3038-4662-9616-55e6dcb34f49-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.298120 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.298039 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ca25d7bb-3038-4662-9616-55e6dcb34f49-secret-telemeter-client\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.298120 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.298068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn9bp\" (UniqueName: \"kubernetes.io/projected/ca25d7bb-3038-4662-9616-55e6dcb34f49-kube-api-access-tn9bp\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.298120 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.298113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca25d7bb-3038-4662-9616-55e6dcb34f49-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.298730 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.298700 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca25d7bb-3038-4662-9616-55e6dcb34f49-serving-certs-ca-bundle\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.298881 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.298735 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca25d7bb-3038-4662-9616-55e6dcb34f49-metrics-client-ca\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.299094 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.299071 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca25d7bb-3038-4662-9616-55e6dcb34f49-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.300474 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.300443 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ca25d7bb-3038-4662-9616-55e6dcb34f49-federate-client-tls\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.300746 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.300724 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ca25d7bb-3038-4662-9616-55e6dcb34f49-secret-telemeter-client\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.300791 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.300769 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ca25d7bb-3038-4662-9616-55e6dcb34f49-telemeter-client-tls\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.300830 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.300814 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca25d7bb-3038-4662-9616-55e6dcb34f49-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.305855 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.305830 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn9bp\" (UniqueName: \"kubernetes.io/projected/ca25d7bb-3038-4662-9616-55e6dcb34f49-kube-api-access-tn9bp\") pod \"telemeter-client-7d5b7988fb-5mbxj\" (UID: \"ca25d7bb-3038-4662-9616-55e6dcb34f49\") " pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.308438 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.308397 2564 generic.go:356] "Generic (PLEG): container finished" podID="d38addb7-debd-46e7-b185-77d46cd891ec" containerID="a80eb8dee9c47dbbd2bacaae342628f54a2cab44524953487ea5437f566bb43a" exitCode=0 Jul 23 00:29:33.308565 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.308502 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d38addb7-debd-46e7-b185-77d46cd891ec","Type":"ContainerDied","Data":"a80eb8dee9c47dbbd2bacaae342628f54a2cab44524953487ea5437f566bb43a"} Jul 23 00:29:33.397055 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.397020 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" Jul 23 00:29:33.532703 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:33.532675 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj"] Jul 23 00:29:33.534661 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:29:33.534639 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca25d7bb_3038_4662_9616_55e6dcb34f49.slice/crio-aef52d7963e7d003e5058765753a75595a3e5431a01db0f542962f7e726310db WatchSource:0}: Error finding container aef52d7963e7d003e5058765753a75595a3e5431a01db0f542962f7e726310db: Status 404 returned error can't find the container with id aef52d7963e7d003e5058765753a75595a3e5431a01db0f542962f7e726310db Jul 23 00:29:34.314843 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:34.314797 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" event={"ID":"ca25d7bb-3038-4662-9616-55e6dcb34f49","Type":"ContainerStarted","Data":"aef52d7963e7d003e5058765753a75595a3e5431a01db0f542962f7e726310db"} Jul 23 00:29:35.321252 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:35.321199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d38addb7-debd-46e7-b185-77d46cd891ec","Type":"ContainerStarted","Data":"3bc648994ecfcadaf32395f30846fd77f956f943c0f820a8483efc871210c232"} Jul 23 00:29:36.328892 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:36.328846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d38addb7-debd-46e7-b185-77d46cd891ec","Type":"ContainerStarted","Data":"21a86a0179c371a1f1deb349608fe739a2c5647c545e4afaf36a949e90054591"} Jul 23 00:29:36.329330 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:36.328938 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d38addb7-debd-46e7-b185-77d46cd891ec","Type":"ContainerStarted","Data":"9a6db8ba8975df34a5550292b317c849494495a0a468d02c43838e177c10cdb0"} Jul 23 00:29:36.329330 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:36.328991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d38addb7-debd-46e7-b185-77d46cd891ec","Type":"ContainerStarted","Data":"d338996d65f868ebe8a83a6c5c3b613b4668ef2664efebcf9bdfafe80ab6b245"} Jul 23 00:29:36.329330 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:36.329031 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d38addb7-debd-46e7-b185-77d46cd891ec","Type":"ContainerStarted","Data":"e6e93230a413ede7fabffad78f37466513b42c02c7c4737cf4b29c5632648d30"} Jul 23 00:29:36.329330 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:36.329078 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d38addb7-debd-46e7-b185-77d46cd891ec","Type":"ContainerStarted","Data":"0d969e02539187b054dcb4ee9c1d31a8cb7a4485a0d39a6765faa3ae10feb989"} Jul 23 00:29:36.330973 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:36.330939 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" event={"ID":"ca25d7bb-3038-4662-9616-55e6dcb34f49","Type":"ContainerStarted","Data":"1fc464634e5afadc36d163672d89dc626e718bf3b0a9d48adffcd9d4b9d7c442"} Jul 23 00:29:36.331081 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:36.330991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" event={"ID":"ca25d7bb-3038-4662-9616-55e6dcb34f49","Type":"ContainerStarted","Data":"87133fd85a2ed0ae1f5959098549f04bdbccd3bc27e2243958a95bb440323404"} Jul 23 00:29:36.331081 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:36.331020 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" event={"ID":"ca25d7bb-3038-4662-9616-55e6dcb34f49","Type":"ContainerStarted","Data":"8b30ba6e86f2c3f20ca164eeb6f971950be50d69fe1f44094092ef00f401c651"} Jul 23 00:29:36.357867 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:36.357799 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.328991817 podStartE2EDuration="6.357779s" totalImagesPullingTime="3.014282901s" totalInitContainerRuntime="14.504282ms" isStatefulPod=false podCreationTimestamp="2026-07-23 00:29:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:29:36.355377679 +0000 UTC m=+368.851046393" watchObservedRunningTime="2026-07-23 00:29:36.357779 +0000 UTC m=+368.853447714" Jul 23 00:29:36.376629 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:36.376577 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7d5b7988fb-5mbxj" podStartSLOduration=1.204855621 podStartE2EDuration="3.376562172s" totalImagesPullingTime="2.171706551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:29:36.374931038 +0000 UTC m=+368.870599768" watchObservedRunningTime="2026-07-23 00:29:36.376562172 +0000 UTC m=+368.872230881" Jul 23 00:29:37.138062 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.138028 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6689bc557-8mnfz"] Jul 23 00:29:37.140583 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.140559 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.150725 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.150697 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6689bc557-8mnfz"] Jul 23 00:29:37.201673 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.201640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:37.229718 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.229682 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64d184c7-7011-41c5-8264-e2b18cd64987-console-serving-cert\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.229718 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.229725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnpgv\" (UniqueName: \"kubernetes.io/projected/64d184c7-7011-41c5-8264-e2b18cd64987-kube-api-access-hnpgv\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.229986 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.229749 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-console-config\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.229986 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.229887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-service-ca\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.229986 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.229928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-trusted-ca-bundle\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.229986 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.229962 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64d184c7-7011-41c5-8264-e2b18cd64987-console-oauth-config\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.230144 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.230010 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-oauth-serving-cert\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.302392 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.302350 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:37.330728 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.330694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-trusted-ca-bundle\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.331190 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.330734 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64d184c7-7011-41c5-8264-e2b18cd64987-console-oauth-config\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.331190 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.330861 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-oauth-serving-cert\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.331190 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.330914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64d184c7-7011-41c5-8264-e2b18cd64987-console-serving-cert\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.331190 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.330938 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnpgv\" (UniqueName: \"kubernetes.io/projected/64d184c7-7011-41c5-8264-e2b18cd64987-kube-api-access-hnpgv\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.331190 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.330961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-console-config\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.331190 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.331023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-service-ca\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.331703 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.331681 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-oauth-serving-cert\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.331827 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.331773 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-service-ca\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.331954 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.331925 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-console-config\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.332161 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.332141 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-trusted-ca-bundle\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.333562 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.333518 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64d184c7-7011-41c5-8264-e2b18cd64987-console-serving-cert\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.333682 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.333669 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64d184c7-7011-41c5-8264-e2b18cd64987-console-oauth-config\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.339518 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.339489 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnpgv\" (UniqueName: \"kubernetes.io/projected/64d184c7-7011-41c5-8264-e2b18cd64987-kube-api-access-hnpgv\") pod \"console-6689bc557-8mnfz\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.399366 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.399268 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:37.451434 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.451389 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:37.586521 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.586491 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6689bc557-8mnfz"] Jul 23 00:29:37.589415 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:29:37.589383 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d184c7_7011_41c5_8264_e2b18cd64987.slice/crio-935a23b8c711ec49ce613059b2fe35e8628a51025fbdb5daf961f0bc0b8fe5cd WatchSource:0}: Error finding container 935a23b8c711ec49ce613059b2fe35e8628a51025fbdb5daf961f0bc0b8fe5cd: Status 404 returned error can't find the container with id 935a23b8c711ec49ce613059b2fe35e8628a51025fbdb5daf961f0bc0b8fe5cd Jul 23 00:29:37.936699 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:37.936665 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:38.343554 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:38.343434 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6689bc557-8mnfz" event={"ID":"64d184c7-7011-41c5-8264-e2b18cd64987","Type":"ContainerStarted","Data":"06d52043c53255e42baab4c8219ead450cd3f24a9114d85024c284338fa23d1d"} Jul 23 00:29:38.343554 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:38.343486 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6689bc557-8mnfz" event={"ID":"64d184c7-7011-41c5-8264-e2b18cd64987","Type":"ContainerStarted","Data":"935a23b8c711ec49ce613059b2fe35e8628a51025fbdb5daf961f0bc0b8fe5cd"} Jul 23 00:29:38.362141 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:38.362071 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6689bc557-8mnfz" podStartSLOduration=1.362050966 podStartE2EDuration="1.362050966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:29:38.3604866 +0000 UTC m=+370.856155351" watchObservedRunningTime="2026-07-23 00:29:38.362050966 +0000 UTC m=+370.857719679" Jul 23 00:29:42.118377 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:42.118328 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:42.219056 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:42.219015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:47.452157 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:47.452109 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:47.452575 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:47.452221 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:47.457207 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:47.457183 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:48.389885 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:48.389856 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:29:48.437493 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:48.437459 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f4bd66545-szh5g"] Jul 23 00:29:48.523044 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:48.523007 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:48.624687 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:48.624653 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:29:48.680885 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:29:48.680799 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:30:13.458937 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.458891 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f4bd66545-szh5g" podUID="957f3964-09d1-44f9-8a13-2070edd225ca" containerName="console" containerID="cri-o://a59cc2f21f4ef353170fbcd4ed48aa6ee38d11fe1eec019f18862f6037b1b742" gracePeriod=15 Jul 23 00:30:13.708561 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.708498 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:30:13.847382 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.847292 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-trusted-ca-bundle\") pod \"957f3964-09d1-44f9-8a13-2070edd225ca\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " Jul 23 00:30:13.847382 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.847347 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-console-config\") pod \"957f3964-09d1-44f9-8a13-2070edd225ca\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " Jul 23 00:30:13.847382 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.847384 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vth4k\" (UniqueName: \"kubernetes.io/projected/957f3964-09d1-44f9-8a13-2070edd225ca-kube-api-access-vth4k\") pod \"957f3964-09d1-44f9-8a13-2070edd225ca\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " Jul 23 00:30:13.847694 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.847413 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-oauth-serving-cert\") pod \"957f3964-09d1-44f9-8a13-2070edd225ca\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " Jul 23 00:30:13.847694 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.847469 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/957f3964-09d1-44f9-8a13-2070edd225ca-console-oauth-config\") pod \"957f3964-09d1-44f9-8a13-2070edd225ca\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " Jul 23 00:30:13.847694 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.847515 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/957f3964-09d1-44f9-8a13-2070edd225ca-console-serving-cert\") pod \"957f3964-09d1-44f9-8a13-2070edd225ca\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " Jul 23 00:30:13.847694 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.847553 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-service-ca\") pod \"957f3964-09d1-44f9-8a13-2070edd225ca\" (UID: \"957f3964-09d1-44f9-8a13-2070edd225ca\") " Jul 23 00:30:13.847895 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.847745 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "957f3964-09d1-44f9-8a13-2070edd225ca" (UID: "957f3964-09d1-44f9-8a13-2070edd225ca"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:30:13.847967 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.847946 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-trusted-ca-bundle\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:30:13.848023 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.847974 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-console-config" (OuterVolumeSpecName: "console-config") pod "957f3964-09d1-44f9-8a13-2070edd225ca" (UID: "957f3964-09d1-44f9-8a13-2070edd225ca"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:30:13.848023 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.847989 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "957f3964-09d1-44f9-8a13-2070edd225ca" (UID: "957f3964-09d1-44f9-8a13-2070edd225ca"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:30:13.848098 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.848038 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-service-ca" (OuterVolumeSpecName: "service-ca") pod "957f3964-09d1-44f9-8a13-2070edd225ca" (UID: "957f3964-09d1-44f9-8a13-2070edd225ca"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:30:13.849791 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.849757 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957f3964-09d1-44f9-8a13-2070edd225ca-kube-api-access-vth4k" (OuterVolumeSpecName: "kube-api-access-vth4k") pod "957f3964-09d1-44f9-8a13-2070edd225ca" (UID: "957f3964-09d1-44f9-8a13-2070edd225ca"). InnerVolumeSpecName "kube-api-access-vth4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:30:13.849927 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.849780 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957f3964-09d1-44f9-8a13-2070edd225ca-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "957f3964-09d1-44f9-8a13-2070edd225ca" (UID: "957f3964-09d1-44f9-8a13-2070edd225ca"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:30:13.849927 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.849857 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957f3964-09d1-44f9-8a13-2070edd225ca-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "957f3964-09d1-44f9-8a13-2070edd225ca" (UID: "957f3964-09d1-44f9-8a13-2070edd225ca"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:30:13.948845 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.948798 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/957f3964-09d1-44f9-8a13-2070edd225ca-console-oauth-config\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:30:13.948845 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.948834 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/957f3964-09d1-44f9-8a13-2070edd225ca-console-serving-cert\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:30:13.948845 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.948844 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:30:13.948845 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.948853 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-console-config\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:30:13.949118 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.948862 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vth4k\" (UniqueName: \"kubernetes.io/projected/957f3964-09d1-44f9-8a13-2070edd225ca-kube-api-access-vth4k\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:30:13.949118 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:13.948871 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/957f3964-09d1-44f9-8a13-2070edd225ca-oauth-serving-cert\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:30:14.494714 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:14.494680 2564 generic.go:356] "Generic (PLEG): container finished" podID="957f3964-09d1-44f9-8a13-2070edd225ca" containerID="a59cc2f21f4ef353170fbcd4ed48aa6ee38d11fe1eec019f18862f6037b1b742" exitCode=0 Jul 23 00:30:14.495119 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:14.494725 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f4bd66545-szh5g" Jul 23 00:30:14.495119 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:14.494763 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4bd66545-szh5g" event={"ID":"957f3964-09d1-44f9-8a13-2070edd225ca","Type":"ContainerDied","Data":"a59cc2f21f4ef353170fbcd4ed48aa6ee38d11fe1eec019f18862f6037b1b742"} Jul 23 00:30:14.495119 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:14.494816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f4bd66545-szh5g" event={"ID":"957f3964-09d1-44f9-8a13-2070edd225ca","Type":"ContainerDied","Data":"50e025d8bba6aeb7552c3349074c8d8500831da311317c26cd13b9c1bb7ce8e6"} Jul 23 00:30:14.495119 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:14.494842 2564 scope.go:117] "RemoveContainer" containerID="a59cc2f21f4ef353170fbcd4ed48aa6ee38d11fe1eec019f18862f6037b1b742" Jul 23 00:30:14.503400 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:14.503380 2564 scope.go:117] "RemoveContainer" containerID="a59cc2f21f4ef353170fbcd4ed48aa6ee38d11fe1eec019f18862f6037b1b742" Jul 23 00:30:14.503709 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:30:14.503687 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59cc2f21f4ef353170fbcd4ed48aa6ee38d11fe1eec019f18862f6037b1b742\": container with ID starting with a59cc2f21f4ef353170fbcd4ed48aa6ee38d11fe1eec019f18862f6037b1b742 not found: ID does not exist" containerID="a59cc2f21f4ef353170fbcd4ed48aa6ee38d11fe1eec019f18862f6037b1b742" Jul 23 00:30:14.503766 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:14.503719 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59cc2f21f4ef353170fbcd4ed48aa6ee38d11fe1eec019f18862f6037b1b742"} err="failed to get container status \"a59cc2f21f4ef353170fbcd4ed48aa6ee38d11fe1eec019f18862f6037b1b742\": rpc error: code = NotFound desc = could not find container \"a59cc2f21f4ef353170fbcd4ed48aa6ee38d11fe1eec019f18862f6037b1b742\": container with ID starting with a59cc2f21f4ef353170fbcd4ed48aa6ee38d11fe1eec019f18862f6037b1b742 not found: ID does not exist" Jul 23 00:30:14.509952 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:14.509918 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f4bd66545-szh5g"] Jul 23 00:30:14.511303 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:14.511281 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f4bd66545-szh5g"] Jul 23 00:30:16.071926 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:16.071893 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="957f3964-09d1-44f9-8a13-2070edd225ca" path="/var/lib/kubelet/pods/957f3964-09d1-44f9-8a13-2070edd225ca/volumes" Jul 23 00:30:21.960504 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:21.960465 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:30:22.012839 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:22.012805 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:30:43.243884 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:43.243845 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:30:43.301099 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:43.301060 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:30:55.188268 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:55.188146 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:30:55.242092 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:30:55.242059 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:31:04.314231 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.314185 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-786675b496-lgplc"] Jul 23 00:31:04.314648 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.314607 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957f3964-09d1-44f9-8a13-2070edd225ca" containerName="console" Jul 23 00:31:04.314648 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.314622 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="957f3964-09d1-44f9-8a13-2070edd225ca" containerName="console" Jul 23 00:31:04.314719 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.314694 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="957f3964-09d1-44f9-8a13-2070edd225ca" containerName="console" Jul 23 00:31:04.317767 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.317746 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.327836 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.327805 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-786675b496-lgplc"] Jul 23 00:31:04.377210 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.377168 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:31:04.378178 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.378154 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-serving-cert\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.378264 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.378192 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-oauth-config\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.378264 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.378228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-service-ca\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.378334 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.378266 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-oauth-serving-cert\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.378334 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.378302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7w9m\" (UniqueName: \"kubernetes.io/projected/6aefe260-fb1f-4495-8d24-1f11d952fe58-kube-api-access-d7w9m\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.378396 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.378334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-trusted-ca-bundle\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.378396 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.378379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-config\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.470885 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.470826 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:31:04.478808 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.478775 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-oauth-serving-cert\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.479012 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.478828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7w9m\" (UniqueName: \"kubernetes.io/projected/6aefe260-fb1f-4495-8d24-1f11d952fe58-kube-api-access-d7w9m\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.479012 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.478860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-trusted-ca-bundle\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.479012 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.478885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-config\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.479012 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.478944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-serving-cert\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.479012 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.478980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-oauth-config\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.479253 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.479016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-service-ca\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.479629 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.479599 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-oauth-serving-cert\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.479754 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.479636 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-config\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.479844 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.479825 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-trusted-ca-bundle\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.479882 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.479833 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-service-ca\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.481450 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.481429 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-serving-cert\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.481542 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.481488 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-oauth-config\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.486887 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.486866 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7w9m\" (UniqueName: \"kubernetes.io/projected/6aefe260-fb1f-4495-8d24-1f11d952fe58-kube-api-access-d7w9m\") pod \"console-786675b496-lgplc\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.525804 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.525776 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:31:04.628048 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.627956 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:04.707734 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.707696 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:31:04.767827 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:04.767798 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-786675b496-lgplc"] Jul 23 00:31:04.770673 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:31:04.770642 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aefe260_fb1f_4495_8d24_1f11d952fe58.slice/crio-cc816fdf3545cbdd1c88c70b5f904b7e01bd6e32aa2dbe783025a249c47737a9 WatchSource:0}: Error finding container cc816fdf3545cbdd1c88c70b5f904b7e01bd6e32aa2dbe783025a249c47737a9: Status 404 returned error can't find the container with id cc816fdf3545cbdd1c88c70b5f904b7e01bd6e32aa2dbe783025a249c47737a9 Jul 23 00:31:05.714243 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:05.714208 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786675b496-lgplc" event={"ID":"6aefe260-fb1f-4495-8d24-1f11d952fe58","Type":"ContainerStarted","Data":"1f074ee1528398706b1f53ead35fb04fe90bc6e7202200133b1f6e80cc352340"} Jul 23 00:31:05.714660 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:05.714256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786675b496-lgplc" event={"ID":"6aefe260-fb1f-4495-8d24-1f11d952fe58","Type":"ContainerStarted","Data":"cc816fdf3545cbdd1c88c70b5f904b7e01bd6e32aa2dbe783025a249c47737a9"} Jul 23 00:31:05.734350 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:05.734293 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-786675b496-lgplc" podStartSLOduration=1.7342686920000001 podStartE2EDuration="1.734268692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:31:05.732413797 +0000 UTC m=+458.228082510" watchObservedRunningTime="2026-07-23 00:31:05.734268692 +0000 UTC m=+458.229937404" Jul 23 00:31:14.628107 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:14.628068 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:14.628513 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:14.628142 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:14.632996 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:14.632974 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:14.758127 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:14.758098 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-786675b496-lgplc" Jul 23 00:31:14.809097 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:14.809064 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6689bc557-8mnfz"] Jul 23 00:31:14.911941 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:14.911904 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:31:15.012473 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:15.012436 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:31:15.113670 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:15.113635 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:31:39.830145 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:39.830100 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6689bc557-8mnfz" podUID="64d184c7-7011-41c5-8264-e2b18cd64987" containerName="console" containerID="cri-o://06d52043c53255e42baab4c8219ead450cd3f24a9114d85024c284338fa23d1d" gracePeriod=15 Jul 23 00:31:40.077735 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.077708 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:31:40.192206 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.192162 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-service-ca\") pod \"64d184c7-7011-41c5-8264-e2b18cd64987\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " Jul 23 00:31:40.192206 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.192213 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-console-config\") pod \"64d184c7-7011-41c5-8264-e2b18cd64987\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " Jul 23 00:31:40.192473 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.192246 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnpgv\" (UniqueName: \"kubernetes.io/projected/64d184c7-7011-41c5-8264-e2b18cd64987-kube-api-access-hnpgv\") pod \"64d184c7-7011-41c5-8264-e2b18cd64987\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " Jul 23 00:31:40.192473 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.192294 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64d184c7-7011-41c5-8264-e2b18cd64987-console-oauth-config\") pod \"64d184c7-7011-41c5-8264-e2b18cd64987\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " Jul 23 00:31:40.192473 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.192322 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64d184c7-7011-41c5-8264-e2b18cd64987-console-serving-cert\") pod \"64d184c7-7011-41c5-8264-e2b18cd64987\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " Jul 23 00:31:40.192473 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.192360 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-trusted-ca-bundle\") pod \"64d184c7-7011-41c5-8264-e2b18cd64987\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " Jul 23 00:31:40.192720 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.192494 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-oauth-serving-cert\") pod \"64d184c7-7011-41c5-8264-e2b18cd64987\" (UID: \"64d184c7-7011-41c5-8264-e2b18cd64987\") " Jul 23 00:31:40.192778 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.192735 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-service-ca" (OuterVolumeSpecName: "service-ca") pod "64d184c7-7011-41c5-8264-e2b18cd64987" (UID: "64d184c7-7011-41c5-8264-e2b18cd64987"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:31:40.192875 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.192829 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-console-config" (OuterVolumeSpecName: "console-config") pod "64d184c7-7011-41c5-8264-e2b18cd64987" (UID: "64d184c7-7011-41c5-8264-e2b18cd64987"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:31:40.192875 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.192836 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:31:40.192875 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.192861 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "64d184c7-7011-41c5-8264-e2b18cd64987" (UID: "64d184c7-7011-41c5-8264-e2b18cd64987"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:31:40.193015 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.192881 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "64d184c7-7011-41c5-8264-e2b18cd64987" (UID: "64d184c7-7011-41c5-8264-e2b18cd64987"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:31:40.194542 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.194510 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d184c7-7011-41c5-8264-e2b18cd64987-kube-api-access-hnpgv" (OuterVolumeSpecName: "kube-api-access-hnpgv") pod "64d184c7-7011-41c5-8264-e2b18cd64987" (UID: "64d184c7-7011-41c5-8264-e2b18cd64987"). InnerVolumeSpecName "kube-api-access-hnpgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:31:40.194618 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.194563 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d184c7-7011-41c5-8264-e2b18cd64987-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "64d184c7-7011-41c5-8264-e2b18cd64987" (UID: "64d184c7-7011-41c5-8264-e2b18cd64987"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:31:40.194656 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.194614 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d184c7-7011-41c5-8264-e2b18cd64987-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "64d184c7-7011-41c5-8264-e2b18cd64987" (UID: "64d184c7-7011-41c5-8264-e2b18cd64987"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:31:40.294006 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.293959 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/64d184c7-7011-41c5-8264-e2b18cd64987-console-serving-cert\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:31:40.294006 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.293994 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-trusted-ca-bundle\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:31:40.294006 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.294004 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-oauth-serving-cert\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:31:40.294006 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.294014 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/64d184c7-7011-41c5-8264-e2b18cd64987-console-config\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:31:40.294006 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.294022 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnpgv\" (UniqueName: \"kubernetes.io/projected/64d184c7-7011-41c5-8264-e2b18cd64987-kube-api-access-hnpgv\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:31:40.294303 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.294031 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/64d184c7-7011-41c5-8264-e2b18cd64987-console-oauth-config\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:31:40.869255 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.869215 2564 generic.go:356] "Generic (PLEG): container finished" podID="64d184c7-7011-41c5-8264-e2b18cd64987" containerID="06d52043c53255e42baab4c8219ead450cd3f24a9114d85024c284338fa23d1d" exitCode=0 Jul 23 00:31:40.869255 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.869256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6689bc557-8mnfz" event={"ID":"64d184c7-7011-41c5-8264-e2b18cd64987","Type":"ContainerDied","Data":"06d52043c53255e42baab4c8219ead450cd3f24a9114d85024c284338fa23d1d"} Jul 23 00:31:40.869255 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.869259 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6689bc557-8mnfz" Jul 23 00:31:40.869806 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.869299 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6689bc557-8mnfz" event={"ID":"64d184c7-7011-41c5-8264-e2b18cd64987","Type":"ContainerDied","Data":"935a23b8c711ec49ce613059b2fe35e8628a51025fbdb5daf961f0bc0b8fe5cd"} Jul 23 00:31:40.869806 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.869319 2564 scope.go:117] "RemoveContainer" containerID="06d52043c53255e42baab4c8219ead450cd3f24a9114d85024c284338fa23d1d" Jul 23 00:31:40.878175 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.878154 2564 scope.go:117] "RemoveContainer" containerID="06d52043c53255e42baab4c8219ead450cd3f24a9114d85024c284338fa23d1d" Jul 23 00:31:40.878450 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:31:40.878430 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d52043c53255e42baab4c8219ead450cd3f24a9114d85024c284338fa23d1d\": container with ID starting with 06d52043c53255e42baab4c8219ead450cd3f24a9114d85024c284338fa23d1d not found: ID does not exist" containerID="06d52043c53255e42baab4c8219ead450cd3f24a9114d85024c284338fa23d1d" Jul 23 00:31:40.878510 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.878460 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d52043c53255e42baab4c8219ead450cd3f24a9114d85024c284338fa23d1d"} err="failed to get container status \"06d52043c53255e42baab4c8219ead450cd3f24a9114d85024c284338fa23d1d\": rpc error: code = NotFound desc = could not find container \"06d52043c53255e42baab4c8219ead450cd3f24a9114d85024c284338fa23d1d\": container with ID starting with 06d52043c53255e42baab4c8219ead450cd3f24a9114d85024c284338fa23d1d not found: ID does not exist" Jul 23 00:31:40.890173 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.890137 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6689bc557-8mnfz"] Jul 23 00:31:40.893658 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:40.893632 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6689bc557-8mnfz"] Jul 23 00:31:42.072375 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:42.072340 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64d184c7-7011-41c5-8264-e2b18cd64987" path="/var/lib/kubelet/pods/64d184c7-7011-41c5-8264-e2b18cd64987/volumes" Jul 23 00:31:43.770839 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:43.770805 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:31:43.870642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:43.870605 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:31:44.441573 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.441518 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw"] Jul 23 00:31:44.441921 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.441907 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64d184c7-7011-41c5-8264-e2b18cd64987" containerName="console" Jul 23 00:31:44.441971 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.441922 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="64d184c7-7011-41c5-8264-e2b18cd64987" containerName="console" Jul 23 00:31:44.442006 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.441990 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="64d184c7-7011-41c5-8264-e2b18cd64987" containerName="console" Jul 23 00:31:44.446631 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.446608 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" Jul 23 00:31:44.449348 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.449318 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 00:31:44.449492 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.449357 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-582rg\"" Jul 23 00:31:44.449800 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.449779 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 00:31:44.454940 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.454913 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw"] Jul 23 00:31:44.531978 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.531942 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/111aa0bf-4d2e-4629-a926-b7922071e590-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw\" (UID: \"111aa0bf-4d2e-4629-a926-b7922071e590\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" Jul 23 00:31:44.532133 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.531987 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/111aa0bf-4d2e-4629-a926-b7922071e590-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw\" (UID: \"111aa0bf-4d2e-4629-a926-b7922071e590\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" Jul 23 00:31:44.532133 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.532021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht2s6\" (UniqueName: \"kubernetes.io/projected/111aa0bf-4d2e-4629-a926-b7922071e590-kube-api-access-ht2s6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw\" (UID: \"111aa0bf-4d2e-4629-a926-b7922071e590\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" Jul 23 00:31:44.632550 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.632497 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/111aa0bf-4d2e-4629-a926-b7922071e590-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw\" (UID: \"111aa0bf-4d2e-4629-a926-b7922071e590\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" Jul 23 00:31:44.632698 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.632584 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/111aa0bf-4d2e-4629-a926-b7922071e590-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw\" (UID: \"111aa0bf-4d2e-4629-a926-b7922071e590\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" Jul 23 00:31:44.632698 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.632615 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht2s6\" (UniqueName: \"kubernetes.io/projected/111aa0bf-4d2e-4629-a926-b7922071e590-kube-api-access-ht2s6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw\" (UID: \"111aa0bf-4d2e-4629-a926-b7922071e590\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" Jul 23 00:31:44.632970 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.632933 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/111aa0bf-4d2e-4629-a926-b7922071e590-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw\" (UID: \"111aa0bf-4d2e-4629-a926-b7922071e590\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" Jul 23 00:31:44.633050 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.632980 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/111aa0bf-4d2e-4629-a926-b7922071e590-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw\" (UID: \"111aa0bf-4d2e-4629-a926-b7922071e590\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" Jul 23 00:31:44.650752 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.650713 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht2s6\" (UniqueName: \"kubernetes.io/projected/111aa0bf-4d2e-4629-a926-b7922071e590-kube-api-access-ht2s6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw\" (UID: \"111aa0bf-4d2e-4629-a926-b7922071e590\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" Jul 23 00:31:44.757951 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.757869 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" Jul 23 00:31:44.897272 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:44.897245 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw"] Jul 23 00:31:44.900066 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:31:44.900033 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111aa0bf_4d2e_4629_a926_b7922071e590.slice/crio-7bad77d92202d6e4f54d954e4627111d6f4d0443b79b36a5c9389d04b236057b WatchSource:0}: Error finding container 7bad77d92202d6e4f54d954e4627111d6f4d0443b79b36a5c9389d04b236057b: Status 404 returned error can't find the container with id 7bad77d92202d6e4f54d954e4627111d6f4d0443b79b36a5c9389d04b236057b Jul 23 00:31:45.892443 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:45.892401 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" event={"ID":"111aa0bf-4d2e-4629-a926-b7922071e590","Type":"ContainerStarted","Data":"7bad77d92202d6e4f54d954e4627111d6f4d0443b79b36a5c9389d04b236057b"} Jul 23 00:31:52.927216 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:52.927179 2564 generic.go:356] "Generic (PLEG): container finished" podID="111aa0bf-4d2e-4629-a926-b7922071e590" containerID="de3ff2a3b95faec6c609040c16350dcc6b02332b544f3a6ee6de5616860c5bdf" exitCode=0 Jul 23 00:31:52.927637 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:52.927222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" event={"ID":"111aa0bf-4d2e-4629-a926-b7922071e590","Type":"ContainerDied","Data":"de3ff2a3b95faec6c609040c16350dcc6b02332b544f3a6ee6de5616860c5bdf"} Jul 23 00:31:55.945097 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:55.945052 2564 generic.go:356] "Generic (PLEG): container finished" podID="111aa0bf-4d2e-4629-a926-b7922071e590" containerID="e0afa125f8b5e13c64c297051ea0d27e3cb5a97d1fdecb25ee8da7db658d9f78" exitCode=0 Jul 23 00:31:55.945499 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:31:55.945133 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" event={"ID":"111aa0bf-4d2e-4629-a926-b7922071e590","Type":"ContainerDied","Data":"e0afa125f8b5e13c64c297051ea0d27e3cb5a97d1fdecb25ee8da7db658d9f78"} Jul 23 00:32:04.993059 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:04.993020 2564 generic.go:356] "Generic (PLEG): container finished" podID="111aa0bf-4d2e-4629-a926-b7922071e590" containerID="a3fce9be9cfb83cfee116ccf903b9d72c91160e7d8e477cc7cd02cba0423ca39" exitCode=0 Jul 23 00:32:04.993463 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:04.993087 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" event={"ID":"111aa0bf-4d2e-4629-a926-b7922071e590","Type":"ContainerDied","Data":"a3fce9be9cfb83cfee116ccf903b9d72c91160e7d8e477cc7cd02cba0423ca39"} Jul 23 00:32:06.122680 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:06.122654 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" Jul 23 00:32:06.224149 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:06.224109 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht2s6\" (UniqueName: \"kubernetes.io/projected/111aa0bf-4d2e-4629-a926-b7922071e590-kube-api-access-ht2s6\") pod \"111aa0bf-4d2e-4629-a926-b7922071e590\" (UID: \"111aa0bf-4d2e-4629-a926-b7922071e590\") " Jul 23 00:32:06.224149 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:06.224156 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/111aa0bf-4d2e-4629-a926-b7922071e590-util\") pod \"111aa0bf-4d2e-4629-a926-b7922071e590\" (UID: \"111aa0bf-4d2e-4629-a926-b7922071e590\") " Jul 23 00:32:06.224386 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:06.224211 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/111aa0bf-4d2e-4629-a926-b7922071e590-bundle\") pod \"111aa0bf-4d2e-4629-a926-b7922071e590\" (UID: \"111aa0bf-4d2e-4629-a926-b7922071e590\") " Jul 23 00:32:06.224943 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:06.224907 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111aa0bf-4d2e-4629-a926-b7922071e590-bundle" (OuterVolumeSpecName: "bundle") pod "111aa0bf-4d2e-4629-a926-b7922071e590" (UID: "111aa0bf-4d2e-4629-a926-b7922071e590"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:32:06.226320 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:06.226292 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111aa0bf-4d2e-4629-a926-b7922071e590-kube-api-access-ht2s6" (OuterVolumeSpecName: "kube-api-access-ht2s6") pod "111aa0bf-4d2e-4629-a926-b7922071e590" (UID: "111aa0bf-4d2e-4629-a926-b7922071e590"). InnerVolumeSpecName "kube-api-access-ht2s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:32:06.228126 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:06.228104 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111aa0bf-4d2e-4629-a926-b7922071e590-util" (OuterVolumeSpecName: "util") pod "111aa0bf-4d2e-4629-a926-b7922071e590" (UID: "111aa0bf-4d2e-4629-a926-b7922071e590"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:32:06.325763 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:06.325655 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/111aa0bf-4d2e-4629-a926-b7922071e590-bundle\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:32:06.325763 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:06.325704 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht2s6\" (UniqueName: \"kubernetes.io/projected/111aa0bf-4d2e-4629-a926-b7922071e590-kube-api-access-ht2s6\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:32:06.325763 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:06.325717 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/111aa0bf-4d2e-4629-a926-b7922071e590-util\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:32:07.002835 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:07.002797 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" event={"ID":"111aa0bf-4d2e-4629-a926-b7922071e590","Type":"ContainerDied","Data":"7bad77d92202d6e4f54d954e4627111d6f4d0443b79b36a5c9389d04b236057b"} Jul 23 00:32:07.002835 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:07.002830 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gp2zw" Jul 23 00:32:07.003052 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:07.002851 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bad77d92202d6e4f54d954e4627111d6f4d0443b79b36a5c9389d04b236057b" Jul 23 00:32:09.677753 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:09.677713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:10.274401 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:10.274353 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:11.634964 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:11.634932 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:12.179182 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.179141 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w2fhv"] Jul 23 00:32:12.179495 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.179483 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="111aa0bf-4d2e-4629-a926-b7922071e590" containerName="extract" Jul 23 00:32:12.179554 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.179497 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="111aa0bf-4d2e-4629-a926-b7922071e590" containerName="extract" Jul 23 00:32:12.179554 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.179520 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="111aa0bf-4d2e-4629-a926-b7922071e590" containerName="util" Jul 23 00:32:12.179554 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.179540 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="111aa0bf-4d2e-4629-a926-b7922071e590" containerName="util" Jul 23 00:32:12.179554 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.179548 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="111aa0bf-4d2e-4629-a926-b7922071e590" containerName="pull" Jul 23 00:32:12.179554 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.179554 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="111aa0bf-4d2e-4629-a926-b7922071e590" containerName="pull" Jul 23 00:32:12.179709 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.179612 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="111aa0bf-4d2e-4629-a926-b7922071e590" containerName="extract" Jul 23 00:32:12.183705 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.183681 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w2fhv" Jul 23 00:32:12.186559 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.186509 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:32:12.186559 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.186550 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-kwd4t\"" Jul 23 00:32:12.186758 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.186521 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 23 00:32:12.199303 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.199272 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w2fhv"] Jul 23 00:32:12.281746 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.281698 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7nbc\" (UniqueName: \"kubernetes.io/projected/47c0d322-da6b-485f-ab04-445e491a8679-kube-api-access-b7nbc\") pod \"cert-manager-operator-controller-manager-55c444bf86-w2fhv\" (UID: \"47c0d322-da6b-485f-ab04-445e491a8679\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w2fhv" Jul 23 00:32:12.281746 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.281746 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47c0d322-da6b-485f-ab04-445e491a8679-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-w2fhv\" (UID: \"47c0d322-da6b-485f-ab04-445e491a8679\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w2fhv" Jul 23 00:32:12.382299 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.382261 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7nbc\" (UniqueName: \"kubernetes.io/projected/47c0d322-da6b-485f-ab04-445e491a8679-kube-api-access-b7nbc\") pod \"cert-manager-operator-controller-manager-55c444bf86-w2fhv\" (UID: \"47c0d322-da6b-485f-ab04-445e491a8679\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w2fhv" Jul 23 00:32:12.382299 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.382298 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47c0d322-da6b-485f-ab04-445e491a8679-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-w2fhv\" (UID: \"47c0d322-da6b-485f-ab04-445e491a8679\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w2fhv" Jul 23 00:32:12.382751 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.382732 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47c0d322-da6b-485f-ab04-445e491a8679-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-w2fhv\" (UID: \"47c0d322-da6b-485f-ab04-445e491a8679\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w2fhv" Jul 23 00:32:12.392803 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.392782 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7nbc\" (UniqueName: \"kubernetes.io/projected/47c0d322-da6b-485f-ab04-445e491a8679-kube-api-access-b7nbc\") pod \"cert-manager-operator-controller-manager-55c444bf86-w2fhv\" (UID: \"47c0d322-da6b-485f-ab04-445e491a8679\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w2fhv" Jul 23 00:32:12.493770 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.493678 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w2fhv" Jul 23 00:32:12.632391 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:12.632242 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w2fhv"] Jul 23 00:32:12.635002 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:32:12.634976 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c0d322_da6b_485f_ab04_445e491a8679.slice/crio-c08cdc95fe14ce2fe6d750ec347d36ecfedbcccbcccdf3490f0c49e37dfaf698 WatchSource:0}: Error finding container c08cdc95fe14ce2fe6d750ec347d36ecfedbcccbcccdf3490f0c49e37dfaf698: Status 404 returned error can't find the container with id c08cdc95fe14ce2fe6d750ec347d36ecfedbcccbcccdf3490f0c49e37dfaf698 Jul 23 00:32:13.030253 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:13.030217 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w2fhv" event={"ID":"47c0d322-da6b-485f-ab04-445e491a8679","Type":"ContainerStarted","Data":"c08cdc95fe14ce2fe6d750ec347d36ecfedbcccbcccdf3490f0c49e37dfaf698"} Jul 23 00:32:13.888448 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:13.888404 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:13.991168 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:13.991118 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:15.041702 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:15.041667 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w2fhv" event={"ID":"47c0d322-da6b-485f-ab04-445e491a8679","Type":"ContainerStarted","Data":"211cdf0114ae53293d41d8a28d5851e272da401842843ce5d9d812b95c9fcbc8"} Jul 23 00:32:15.066997 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:15.066930 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w2fhv" podStartSLOduration=1.351410643 podStartE2EDuration="3.06690543s" totalImagesPullingTime="1.715494787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:32:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:32:15.06337883 +0000 UTC m=+527.559047543" watchObservedRunningTime="2026-07-23 00:32:15.06690543 +0000 UTC m=+527.562574144" Jul 23 00:32:15.444221 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:15.444173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:15.556602 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:15.556561 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:16.905479 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:16.905434 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv"] Jul 23 00:32:16.908623 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:16.908597 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" Jul 23 00:32:16.911247 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:16.911223 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 00:32:16.912317 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:16.912294 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 00:32:16.912432 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:16.912316 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-582rg\"" Jul 23 00:32:16.916331 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:16.916298 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv"] Jul 23 00:32:17.024512 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:17.024473 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6e11b5e-fd7a-4c3c-8166-38358caecb06-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv\" (UID: \"f6e11b5e-fd7a-4c3c-8166-38358caecb06\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" Jul 23 00:32:17.024762 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:17.024556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6e11b5e-fd7a-4c3c-8166-38358caecb06-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv\" (UID: \"f6e11b5e-fd7a-4c3c-8166-38358caecb06\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" Jul 23 00:32:17.024762 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:17.024696 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wm8p\" (UniqueName: \"kubernetes.io/projected/f6e11b5e-fd7a-4c3c-8166-38358caecb06-kube-api-access-5wm8p\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv\" (UID: \"f6e11b5e-fd7a-4c3c-8166-38358caecb06\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" Jul 23 00:32:17.126193 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:17.126152 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6e11b5e-fd7a-4c3c-8166-38358caecb06-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv\" (UID: \"f6e11b5e-fd7a-4c3c-8166-38358caecb06\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" Jul 23 00:32:17.126193 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:17.126205 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6e11b5e-fd7a-4c3c-8166-38358caecb06-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv\" (UID: \"f6e11b5e-fd7a-4c3c-8166-38358caecb06\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" Jul 23 00:32:17.126418 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:17.126249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wm8p\" (UniqueName: \"kubernetes.io/projected/f6e11b5e-fd7a-4c3c-8166-38358caecb06-kube-api-access-5wm8p\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv\" (UID: \"f6e11b5e-fd7a-4c3c-8166-38358caecb06\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" Jul 23 00:32:17.126694 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:17.126668 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6e11b5e-fd7a-4c3c-8166-38358caecb06-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv\" (UID: \"f6e11b5e-fd7a-4c3c-8166-38358caecb06\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" Jul 23 00:32:17.126733 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:17.126680 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6e11b5e-fd7a-4c3c-8166-38358caecb06-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv\" (UID: \"f6e11b5e-fd7a-4c3c-8166-38358caecb06\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" Jul 23 00:32:17.135103 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:17.135067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wm8p\" (UniqueName: \"kubernetes.io/projected/f6e11b5e-fd7a-4c3c-8166-38358caecb06-kube-api-access-5wm8p\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv\" (UID: \"f6e11b5e-fd7a-4c3c-8166-38358caecb06\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" Jul 23 00:32:17.219410 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:17.219330 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" Jul 23 00:32:17.354386 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:17.354357 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv"] Jul 23 00:32:17.356250 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:32:17.356223 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e11b5e_fd7a_4c3c_8166_38358caecb06.slice/crio-922398c5ac9187a79614028a9c9b955cf21af12eb8b9a82cb52f871cdf972415 WatchSource:0}: Error finding container 922398c5ac9187a79614028a9c9b955cf21af12eb8b9a82cb52f871cdf972415: Status 404 returned error can't find the container with id 922398c5ac9187a79614028a9c9b955cf21af12eb8b9a82cb52f871cdf972415 Jul 23 00:32:17.582787 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:17.582739 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:17.987832 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:17.987798 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:18.058556 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:18.058516 2564 generic.go:356] "Generic (PLEG): container finished" podID="f6e11b5e-fd7a-4c3c-8166-38358caecb06" containerID="a81d599628deb0515d1fdc56a851481179019a695f656eeebd11b691385c1aa4" exitCode=0 Jul 23 00:32:18.058748 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:18.058602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" event={"ID":"f6e11b5e-fd7a-4c3c-8166-38358caecb06","Type":"ContainerDied","Data":"a81d599628deb0515d1fdc56a851481179019a695f656eeebd11b691385c1aa4"} Jul 23 00:32:18.058748 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:18.058647 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" event={"ID":"f6e11b5e-fd7a-4c3c-8166-38358caecb06","Type":"ContainerStarted","Data":"922398c5ac9187a79614028a9c9b955cf21af12eb8b9a82cb52f871cdf972415"} Jul 23 00:32:18.952793 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:18.952757 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bqqt9"] Jul 23 00:32:18.955743 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:18.955717 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bqqt9" Jul 23 00:32:18.958381 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:18.958353 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 23 00:32:18.958516 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:18.958363 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 23 00:32:18.959663 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:18.959640 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-hqw2q\"" Jul 23 00:32:18.963350 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:18.963320 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bqqt9"] Jul 23 00:32:19.044007 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:19.043957 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22k5j\" (UniqueName: \"kubernetes.io/projected/fa69af6c-b0fe-48c7-8fa9-fcfa8864ad5f-kube-api-access-22k5j\") pod \"cert-manager-cainjector-559cdd8c87-bqqt9\" (UID: \"fa69af6c-b0fe-48c7-8fa9-fcfa8864ad5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bqqt9" Jul 23 00:32:19.044499 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:19.044035 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa69af6c-b0fe-48c7-8fa9-fcfa8864ad5f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bqqt9\" (UID: \"fa69af6c-b0fe-48c7-8fa9-fcfa8864ad5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bqqt9" Jul 23 00:32:19.145272 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:19.145233 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22k5j\" (UniqueName: \"kubernetes.io/projected/fa69af6c-b0fe-48c7-8fa9-fcfa8864ad5f-kube-api-access-22k5j\") pod \"cert-manager-cainjector-559cdd8c87-bqqt9\" (UID: \"fa69af6c-b0fe-48c7-8fa9-fcfa8864ad5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bqqt9" Jul 23 00:32:19.145448 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:19.145285 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa69af6c-b0fe-48c7-8fa9-fcfa8864ad5f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bqqt9\" (UID: \"fa69af6c-b0fe-48c7-8fa9-fcfa8864ad5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bqqt9" Jul 23 00:32:19.154850 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:19.154822 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa69af6c-b0fe-48c7-8fa9-fcfa8864ad5f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-bqqt9\" (UID: \"fa69af6c-b0fe-48c7-8fa9-fcfa8864ad5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bqqt9" Jul 23 00:32:19.155070 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:19.155047 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22k5j\" (UniqueName: \"kubernetes.io/projected/fa69af6c-b0fe-48c7-8fa9-fcfa8864ad5f-kube-api-access-22k5j\") pod \"cert-manager-cainjector-559cdd8c87-bqqt9\" (UID: \"fa69af6c-b0fe-48c7-8fa9-fcfa8864ad5f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-bqqt9" Jul 23 00:32:19.276429 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:19.276335 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bqqt9" Jul 23 00:32:19.430336 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:19.430236 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-bqqt9"] Jul 23 00:32:20.002684 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:32:20.002648 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa69af6c_b0fe_48c7_8fa9_fcfa8864ad5f.slice/crio-843dab2bc831c7125d1d36bd7685c9d811a2a78341b18d6bee50526d00d4c923 WatchSource:0}: Error finding container 843dab2bc831c7125d1d36bd7685c9d811a2a78341b18d6bee50526d00d4c923: Status 404 returned error can't find the container with id 843dab2bc831c7125d1d36bd7685c9d811a2a78341b18d6bee50526d00d4c923 Jul 23 00:32:20.074439 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:20.074297 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bqqt9" event={"ID":"fa69af6c-b0fe-48c7-8fa9-fcfa8864ad5f","Type":"ContainerStarted","Data":"843dab2bc831c7125d1d36bd7685c9d811a2a78341b18d6bee50526d00d4c923"} Jul 23 00:32:20.676568 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:20.676510 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:20.735366 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:20.735259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:21.078930 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:21.078840 2564 generic.go:356] "Generic (PLEG): container finished" podID="f6e11b5e-fd7a-4c3c-8166-38358caecb06" containerID="c1d4eeb156cfd455cde000198dae053f3334ff78ee96866db9d4078bf806bcc3" exitCode=0 Jul 23 00:32:21.078930 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:21.078901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" event={"ID":"f6e11b5e-fd7a-4c3c-8166-38358caecb06","Type":"ContainerDied","Data":"c1d4eeb156cfd455cde000198dae053f3334ff78ee96866db9d4078bf806bcc3"} Jul 23 00:32:22.062546 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:22.062488 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:22.087028 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:22.086986 2564 generic.go:356] "Generic (PLEG): container finished" podID="f6e11b5e-fd7a-4c3c-8166-38358caecb06" containerID="bc10ae1d1dbebafc366b539a4f5cab205ba151ef415a34d1cb453f21ddb7a281" exitCode=0 Jul 23 00:32:22.087465 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:22.087043 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" event={"ID":"f6e11b5e-fd7a-4c3c-8166-38358caecb06","Type":"ContainerDied","Data":"bc10ae1d1dbebafc366b539a4f5cab205ba151ef415a34d1cb453f21ddb7a281"} Jul 23 00:32:23.093663 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.093555 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bqqt9" event={"ID":"fa69af6c-b0fe-48c7-8fa9-fcfa8864ad5f","Type":"ContainerStarted","Data":"1c4576778c436f8d5f0635226bf6db70a8d09d11f69157b644454dd1a773fb4f"} Jul 23 00:32:23.111171 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.111118 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-bqqt9" podStartSLOduration=2.45209472 podStartE2EDuration="5.111102674s" totalImagesPullingTime="2.659007954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:32:23.109171524 +0000 UTC m=+535.604840239" watchObservedRunningTime="2026-07-23 00:32:23.111102674 +0000 UTC m=+535.606771388" Jul 23 00:32:23.259768 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.259729 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" Jul 23 00:32:23.285707 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.285672 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wm8p\" (UniqueName: \"kubernetes.io/projected/f6e11b5e-fd7a-4c3c-8166-38358caecb06-kube-api-access-5wm8p\") pod \"f6e11b5e-fd7a-4c3c-8166-38358caecb06\" (UID: \"f6e11b5e-fd7a-4c3c-8166-38358caecb06\") " Jul 23 00:32:23.285899 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.285763 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6e11b5e-fd7a-4c3c-8166-38358caecb06-util\") pod \"f6e11b5e-fd7a-4c3c-8166-38358caecb06\" (UID: \"f6e11b5e-fd7a-4c3c-8166-38358caecb06\") " Jul 23 00:32:23.285899 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.285781 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6e11b5e-fd7a-4c3c-8166-38358caecb06-bundle\") pod \"f6e11b5e-fd7a-4c3c-8166-38358caecb06\" (UID: \"f6e11b5e-fd7a-4c3c-8166-38358caecb06\") " Jul 23 00:32:23.286211 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.286184 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e11b5e-fd7a-4c3c-8166-38358caecb06-bundle" (OuterVolumeSpecName: "bundle") pod "f6e11b5e-fd7a-4c3c-8166-38358caecb06" (UID: "f6e11b5e-fd7a-4c3c-8166-38358caecb06"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:32:23.287891 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.287856 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e11b5e-fd7a-4c3c-8166-38358caecb06-kube-api-access-5wm8p" (OuterVolumeSpecName: "kube-api-access-5wm8p") pod "f6e11b5e-fd7a-4c3c-8166-38358caecb06" (UID: "f6e11b5e-fd7a-4c3c-8166-38358caecb06"). InnerVolumeSpecName "kube-api-access-5wm8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:32:23.301911 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.301876 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e11b5e-fd7a-4c3c-8166-38358caecb06-util" (OuterVolumeSpecName: "util") pod "f6e11b5e-fd7a-4c3c-8166-38358caecb06" (UID: "f6e11b5e-fd7a-4c3c-8166-38358caecb06"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:32:23.386729 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.386698 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wm8p\" (UniqueName: \"kubernetes.io/projected/f6e11b5e-fd7a-4c3c-8166-38358caecb06-kube-api-access-5wm8p\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:32:23.386729 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.386730 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6e11b5e-fd7a-4c3c-8166-38358caecb06-util\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:32:23.386908 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.386739 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6e11b5e-fd7a-4c3c-8166-38358caecb06-bundle\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:32:23.573676 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.573638 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-w9qdm"] Jul 23 00:32:23.574037 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.574022 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e11b5e-fd7a-4c3c-8166-38358caecb06" containerName="extract" Jul 23 00:32:23.574089 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.574038 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e11b5e-fd7a-4c3c-8166-38358caecb06" containerName="extract" Jul 23 00:32:23.574089 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.574051 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e11b5e-fd7a-4c3c-8166-38358caecb06" containerName="util" Jul 23 00:32:23.574089 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.574056 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e11b5e-fd7a-4c3c-8166-38358caecb06" containerName="util" Jul 23 00:32:23.574089 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.574065 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e11b5e-fd7a-4c3c-8166-38358caecb06" containerName="pull" Jul 23 00:32:23.574089 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.574071 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e11b5e-fd7a-4c3c-8166-38358caecb06" containerName="pull" Jul 23 00:32:23.574261 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.574132 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e11b5e-fd7a-4c3c-8166-38358caecb06" containerName="extract" Jul 23 00:32:23.576612 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.576593 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-w9qdm" Jul 23 00:32:23.579224 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.579199 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-64sxm\"" Jul 23 00:32:23.586175 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.586142 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-w9qdm"] Jul 23 00:32:23.689409 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.689314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s55m\" (UniqueName: \"kubernetes.io/projected/00a8ee9a-0e21-4198-9f01-b90d467179ea-kube-api-access-2s55m\") pod \"cert-manager-webhook-6486ffd997-w9qdm\" (UID: \"00a8ee9a-0e21-4198-9f01-b90d467179ea\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w9qdm" Jul 23 00:32:23.689409 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.689408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00a8ee9a-0e21-4198-9f01-b90d467179ea-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-w9qdm\" (UID: \"00a8ee9a-0e21-4198-9f01-b90d467179ea\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w9qdm" Jul 23 00:32:23.790462 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.790398 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00a8ee9a-0e21-4198-9f01-b90d467179ea-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-w9qdm\" (UID: \"00a8ee9a-0e21-4198-9f01-b90d467179ea\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w9qdm" Jul 23 00:32:23.790707 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.790509 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s55m\" (UniqueName: \"kubernetes.io/projected/00a8ee9a-0e21-4198-9f01-b90d467179ea-kube-api-access-2s55m\") pod \"cert-manager-webhook-6486ffd997-w9qdm\" (UID: \"00a8ee9a-0e21-4198-9f01-b90d467179ea\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w9qdm" Jul 23 00:32:23.799096 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.799056 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00a8ee9a-0e21-4198-9f01-b90d467179ea-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-w9qdm\" (UID: \"00a8ee9a-0e21-4198-9f01-b90d467179ea\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w9qdm" Jul 23 00:32:23.799286 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.799266 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s55m\" (UniqueName: \"kubernetes.io/projected/00a8ee9a-0e21-4198-9f01-b90d467179ea-kube-api-access-2s55m\") pod \"cert-manager-webhook-6486ffd997-w9qdm\" (UID: \"00a8ee9a-0e21-4198-9f01-b90d467179ea\") " pod="cert-manager/cert-manager-webhook-6486ffd997-w9qdm" Jul 23 00:32:23.886639 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:23.886597 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-w9qdm" Jul 23 00:32:24.027539 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:24.027476 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-w9qdm"] Jul 23 00:32:24.029867 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:32:24.029834 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00a8ee9a_0e21_4198_9f01_b90d467179ea.slice/crio-d66e0107f5a9486231f97bc74dba33da2c6d007360392840f933c06d06729691 WatchSource:0}: Error finding container d66e0107f5a9486231f97bc74dba33da2c6d007360392840f933c06d06729691: Status 404 returned error can't find the container with id d66e0107f5a9486231f97bc74dba33da2c6d007360392840f933c06d06729691 Jul 23 00:32:24.101229 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:24.101198 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-w9qdm" event={"ID":"00a8ee9a-0e21-4198-9f01-b90d467179ea","Type":"ContainerStarted","Data":"d66e0107f5a9486231f97bc74dba33da2c6d007360392840f933c06d06729691"} Jul 23 00:32:24.103336 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:24.103312 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" Jul 23 00:32:24.103336 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:24.103315 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzclrv" event={"ID":"f6e11b5e-fd7a-4c3c-8166-38358caecb06","Type":"ContainerDied","Data":"922398c5ac9187a79614028a9c9b955cf21af12eb8b9a82cb52f871cdf972415"} Jul 23 00:32:24.103513 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:24.103370 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="922398c5ac9187a79614028a9c9b955cf21af12eb8b9a82cb52f871cdf972415" Jul 23 00:32:24.748334 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:24.748294 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:24.800678 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:24.800633 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:25.109827 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:25.109727 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-w9qdm" event={"ID":"00a8ee9a-0e21-4198-9f01-b90d467179ea","Type":"ContainerStarted","Data":"17d10f7e32e3f22706be05375870d815249730080c2628bd03838556f54fe6a7"} Jul 23 00:32:25.110207 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:25.109862 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-w9qdm" Jul 23 00:32:25.125636 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:25.125580 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-w9qdm" podStartSLOduration=2.125559852 podStartE2EDuration="2.125559852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:32:25.124214022 +0000 UTC m=+537.619882736" watchObservedRunningTime="2026-07-23 00:32:25.125559852 +0000 UTC m=+537.621228566" Jul 23 00:32:26.294439 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:26.294398 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:31.117247 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:31.117212 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-w9qdm" Jul 23 00:32:32.090448 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:32.090354 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:32.189301 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:32.189265 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:36.055031 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:36.054997 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nlkkh"] Jul 23 00:32:36.097212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:36.097180 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nlkkh"] Jul 23 00:32:36.097373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:36.097300 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nlkkh" Jul 23 00:32:36.099882 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:36.099852 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qzc5g\"" Jul 23 00:32:36.201464 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:36.201419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f2d9b26-3cd9-4169-8cca-79cdca9299af-bound-sa-token\") pod \"cert-manager-55b4d954d5-nlkkh\" (UID: \"1f2d9b26-3cd9-4169-8cca-79cdca9299af\") " pod="cert-manager/cert-manager-55b4d954d5-nlkkh" Jul 23 00:32:36.201656 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:36.201520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4j8m\" (UniqueName: \"kubernetes.io/projected/1f2d9b26-3cd9-4169-8cca-79cdca9299af-kube-api-access-m4j8m\") pod \"cert-manager-55b4d954d5-nlkkh\" (UID: \"1f2d9b26-3cd9-4169-8cca-79cdca9299af\") " pod="cert-manager/cert-manager-55b4d954d5-nlkkh" Jul 23 00:32:36.303058 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:36.303014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4j8m\" (UniqueName: \"kubernetes.io/projected/1f2d9b26-3cd9-4169-8cca-79cdca9299af-kube-api-access-m4j8m\") pod \"cert-manager-55b4d954d5-nlkkh\" (UID: \"1f2d9b26-3cd9-4169-8cca-79cdca9299af\") " pod="cert-manager/cert-manager-55b4d954d5-nlkkh" Jul 23 00:32:36.303288 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:36.303079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f2d9b26-3cd9-4169-8cca-79cdca9299af-bound-sa-token\") pod \"cert-manager-55b4d954d5-nlkkh\" (UID: \"1f2d9b26-3cd9-4169-8cca-79cdca9299af\") " pod="cert-manager/cert-manager-55b4d954d5-nlkkh" Jul 23 00:32:36.311149 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:36.311067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4j8m\" (UniqueName: \"kubernetes.io/projected/1f2d9b26-3cd9-4169-8cca-79cdca9299af-kube-api-access-m4j8m\") pod \"cert-manager-55b4d954d5-nlkkh\" (UID: \"1f2d9b26-3cd9-4169-8cca-79cdca9299af\") " pod="cert-manager/cert-manager-55b4d954d5-nlkkh" Jul 23 00:32:36.311149 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:36.311068 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1f2d9b26-3cd9-4169-8cca-79cdca9299af-bound-sa-token\") pod \"cert-manager-55b4d954d5-nlkkh\" (UID: \"1f2d9b26-3cd9-4169-8cca-79cdca9299af\") " pod="cert-manager/cert-manager-55b4d954d5-nlkkh" Jul 23 00:32:36.407757 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:36.407719 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-nlkkh" Jul 23 00:32:36.572453 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:36.572364 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-nlkkh"] Jul 23 00:32:36.575891 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:32:36.575860 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f2d9b26_3cd9_4169_8cca_79cdca9299af.slice/crio-e994b22f5077a19d6df8b9b24989bae5a49a8fe51e64c813121b9b1943b51fc8 WatchSource:0}: Error finding container e994b22f5077a19d6df8b9b24989bae5a49a8fe51e64c813121b9b1943b51fc8: Status 404 returned error can't find the container with id e994b22f5077a19d6df8b9b24989bae5a49a8fe51e64c813121b9b1943b51fc8 Jul 23 00:32:37.170687 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:37.170650 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nlkkh" event={"ID":"1f2d9b26-3cd9-4169-8cca-79cdca9299af","Type":"ContainerStarted","Data":"3c3c0c1115b5ac2d200698cfb34d901c2be7b237ef9a3cf7dd55719549df9530"} Jul 23 00:32:37.171075 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:37.170693 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-nlkkh" event={"ID":"1f2d9b26-3cd9-4169-8cca-79cdca9299af","Type":"ContainerStarted","Data":"e994b22f5077a19d6df8b9b24989bae5a49a8fe51e64c813121b9b1943b51fc8"} Jul 23 00:32:37.186880 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:37.186830 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-nlkkh" podStartSLOduration=1.1868106250000001 podStartE2EDuration="1.186810625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:32:37.186427996 +0000 UTC m=+549.682096709" watchObservedRunningTime="2026-07-23 00:32:37.186810625 +0000 UTC m=+549.682479339" Jul 23 00:32:38.404494 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:38.404463 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:38.505265 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:38.505225 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:40.128712 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:40.128669 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:40.948050 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:40.948010 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z"] Jul 23 00:32:40.950693 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:40.950674 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" Jul 23 00:32:40.953247 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:40.953227 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 00:32:40.954334 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:40.954318 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 00:32:40.954384 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:40.954344 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-582rg\"" Jul 23 00:32:40.960144 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:40.960117 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z"] Jul 23 00:32:41.045203 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:41.045161 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z\" (UID: \"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" Jul 23 00:32:41.045355 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:41.045273 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcdfd\" (UniqueName: \"kubernetes.io/projected/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-kube-api-access-bcdfd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z\" (UID: \"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" Jul 23 00:32:41.045355 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:41.045296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z\" (UID: \"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" Jul 23 00:32:41.146719 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:41.146668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z\" (UID: \"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" Jul 23 00:32:41.147082 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:41.146785 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcdfd\" (UniqueName: \"kubernetes.io/projected/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-kube-api-access-bcdfd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z\" (UID: \"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" Jul 23 00:32:41.147082 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:41.146808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z\" (UID: \"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" Jul 23 00:32:41.147082 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:41.147065 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z\" (UID: \"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" Jul 23 00:32:41.147210 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:41.147156 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z\" (UID: \"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" Jul 23 00:32:41.155884 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:41.155861 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcdfd\" (UniqueName: \"kubernetes.io/projected/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-kube-api-access-bcdfd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z\" (UID: \"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" Jul 23 00:32:41.262048 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:41.261970 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" Jul 23 00:32:41.396373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:41.396348 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z"] Jul 23 00:32:41.397867 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:32:41.397840 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc049d36_f1d9_4cb0_8786_9d7bfd9978b4.slice/crio-cb82a4bdc198aabf89ea293413acbb81c281b567a88ff915ea66ec83c0692867 WatchSource:0}: Error finding container cb82a4bdc198aabf89ea293413acbb81c281b567a88ff915ea66ec83c0692867: Status 404 returned error can't find the container with id cb82a4bdc198aabf89ea293413acbb81c281b567a88ff915ea66ec83c0692867 Jul 23 00:32:42.199983 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:42.199948 2564 generic.go:356] "Generic (PLEG): container finished" podID="cc049d36-f1d9-4cb0-8786-9d7bfd9978b4" containerID="1b02b8d1267235fd7fde5ac75909e7420773b0f65a820ad5eeb358e5e96a6466" exitCode=0 Jul 23 00:32:42.200443 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:42.200047 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" event={"ID":"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4","Type":"ContainerDied","Data":"1b02b8d1267235fd7fde5ac75909e7420773b0f65a820ad5eeb358e5e96a6466"} Jul 23 00:32:42.200443 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:42.200111 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" event={"ID":"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4","Type":"ContainerStarted","Data":"cb82a4bdc198aabf89ea293413acbb81c281b567a88ff915ea66ec83c0692867"} Jul 23 00:32:43.207142 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:43.207104 2564 generic.go:356] "Generic (PLEG): container finished" podID="cc049d36-f1d9-4cb0-8786-9d7bfd9978b4" containerID="87327de24b95a16a42c7c125bd994be0877ca813ffe16c80571372e234aeba9d" exitCode=0 Jul 23 00:32:43.207575 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:43.207186 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" event={"ID":"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4","Type":"ContainerDied","Data":"87327de24b95a16a42c7c125bd994be0877ca813ffe16c80571372e234aeba9d"} Jul 23 00:32:44.213856 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:44.213812 2564 generic.go:356] "Generic (PLEG): container finished" podID="cc049d36-f1d9-4cb0-8786-9d7bfd9978b4" containerID="0ee0d62c9e16bee22730acc7ca10be63ca6e99d8b3974654461a7b1de41603dc" exitCode=0 Jul 23 00:32:44.214285 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:44.213880 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" event={"ID":"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4","Type":"ContainerDied","Data":"0ee0d62c9e16bee22730acc7ca10be63ca6e99d8b3974654461a7b1de41603dc"} Jul 23 00:32:45.347822 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:45.347799 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" Jul 23 00:32:45.486598 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:45.486472 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-bundle\") pod \"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4\" (UID: \"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4\") " Jul 23 00:32:45.486598 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:45.486573 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-util\") pod \"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4\" (UID: \"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4\") " Jul 23 00:32:45.486823 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:45.486611 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcdfd\" (UniqueName: \"kubernetes.io/projected/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-kube-api-access-bcdfd\") pod \"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4\" (UID: \"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4\") " Jul 23 00:32:45.487244 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:45.487215 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-bundle" (OuterVolumeSpecName: "bundle") pod "cc049d36-f1d9-4cb0-8786-9d7bfd9978b4" (UID: "cc049d36-f1d9-4cb0-8786-9d7bfd9978b4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:32:45.488782 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:45.488749 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-kube-api-access-bcdfd" (OuterVolumeSpecName: "kube-api-access-bcdfd") pod "cc049d36-f1d9-4cb0-8786-9d7bfd9978b4" (UID: "cc049d36-f1d9-4cb0-8786-9d7bfd9978b4"). InnerVolumeSpecName "kube-api-access-bcdfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:32:45.492023 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:45.491998 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-util" (OuterVolumeSpecName: "util") pod "cc049d36-f1d9-4cb0-8786-9d7bfd9978b4" (UID: "cc049d36-f1d9-4cb0-8786-9d7bfd9978b4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:32:45.587836 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:45.587795 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-bundle\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:32:45.587836 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:45.587829 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-util\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:32:45.587836 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:45.587838 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcdfd\" (UniqueName: \"kubernetes.io/projected/cc049d36-f1d9-4cb0-8786-9d7bfd9978b4-kube-api-access-bcdfd\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:32:46.228632 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:46.228600 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" Jul 23 00:32:46.228632 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:46.228613 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aft6g8z" event={"ID":"cc049d36-f1d9-4cb0-8786-9d7bfd9978b4","Type":"ContainerDied","Data":"cb82a4bdc198aabf89ea293413acbb81c281b567a88ff915ea66ec83c0692867"} Jul 23 00:32:46.228841 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:46.228663 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb82a4bdc198aabf89ea293413acbb81c281b567a88ff915ea66ec83c0692867" Jul 23 00:32:54.700697 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:54.700663 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:32:55.690242 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.690205 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r"] Jul 23 00:32:55.690628 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.690614 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc049d36-f1d9-4cb0-8786-9d7bfd9978b4" containerName="extract" Jul 23 00:32:55.690726 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.690630 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc049d36-f1d9-4cb0-8786-9d7bfd9978b4" containerName="extract" Jul 23 00:32:55.690726 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.690649 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc049d36-f1d9-4cb0-8786-9d7bfd9978b4" containerName="pull" Jul 23 00:32:55.690726 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.690655 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc049d36-f1d9-4cb0-8786-9d7bfd9978b4" containerName="pull" Jul 23 00:32:55.690726 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.690674 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc049d36-f1d9-4cb0-8786-9d7bfd9978b4" containerName="util" Jul 23 00:32:55.690726 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.690679 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc049d36-f1d9-4cb0-8786-9d7bfd9978b4" containerName="util" Jul 23 00:32:55.690866 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.690731 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc049d36-f1d9-4cb0-8786-9d7bfd9978b4" containerName="extract" Jul 23 00:32:55.693214 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.693194 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:55.696335 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.696305 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 00:32:55.696335 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.696330 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hhnpz\"" Jul 23 00:32:55.696604 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.696309 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 00:32:55.696685 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.696647 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 00:32:55.696754 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.696725 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 00:32:55.732546 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.732489 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r"] Jul 23 00:32:55.879218 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.879173 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/546b7cfd-e56a-413e-b157-8a1810969aec-tmp\") pod \"opendatahub-operator-controller-manager-695645577-b6l8r\" (UID: \"546b7cfd-e56a-413e-b157-8a1810969aec\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:55.879382 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.879243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/546b7cfd-e56a-413e-b157-8a1810969aec-webhook-cert\") pod \"opendatahub-operator-controller-manager-695645577-b6l8r\" (UID: \"546b7cfd-e56a-413e-b157-8a1810969aec\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:55.879382 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.879270 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4zm4\" (UniqueName: \"kubernetes.io/projected/546b7cfd-e56a-413e-b157-8a1810969aec-kube-api-access-v4zm4\") pod \"opendatahub-operator-controller-manager-695645577-b6l8r\" (UID: \"546b7cfd-e56a-413e-b157-8a1810969aec\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:55.879382 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.879345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/546b7cfd-e56a-413e-b157-8a1810969aec-manifests\") pod \"opendatahub-operator-controller-manager-695645577-b6l8r\" (UID: \"546b7cfd-e56a-413e-b157-8a1810969aec\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:55.879382 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.879368 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/546b7cfd-e56a-413e-b157-8a1810969aec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-695645577-b6l8r\" (UID: \"546b7cfd-e56a-413e-b157-8a1810969aec\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:55.980654 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.980519 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/546b7cfd-e56a-413e-b157-8a1810969aec-tmp\") pod \"opendatahub-operator-controller-manager-695645577-b6l8r\" (UID: \"546b7cfd-e56a-413e-b157-8a1810969aec\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:55.980654 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.980644 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/546b7cfd-e56a-413e-b157-8a1810969aec-webhook-cert\") pod \"opendatahub-operator-controller-manager-695645577-b6l8r\" (UID: \"546b7cfd-e56a-413e-b157-8a1810969aec\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:55.980863 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.980676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4zm4\" (UniqueName: \"kubernetes.io/projected/546b7cfd-e56a-413e-b157-8a1810969aec-kube-api-access-v4zm4\") pod \"opendatahub-operator-controller-manager-695645577-b6l8r\" (UID: \"546b7cfd-e56a-413e-b157-8a1810969aec\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:55.980863 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.980765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/546b7cfd-e56a-413e-b157-8a1810969aec-manifests\") pod \"opendatahub-operator-controller-manager-695645577-b6l8r\" (UID: \"546b7cfd-e56a-413e-b157-8a1810969aec\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:55.980863 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.980795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/546b7cfd-e56a-413e-b157-8a1810969aec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-695645577-b6l8r\" (UID: \"546b7cfd-e56a-413e-b157-8a1810969aec\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:55.981134 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.981109 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/546b7cfd-e56a-413e-b157-8a1810969aec-manifests\") pod \"opendatahub-operator-controller-manager-695645577-b6l8r\" (UID: \"546b7cfd-e56a-413e-b157-8a1810969aec\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:55.983075 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.983038 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/546b7cfd-e56a-413e-b157-8a1810969aec-tmp\") pod \"opendatahub-operator-controller-manager-695645577-b6l8r\" (UID: \"546b7cfd-e56a-413e-b157-8a1810969aec\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:55.983197 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.983157 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/546b7cfd-e56a-413e-b157-8a1810969aec-webhook-cert\") pod \"opendatahub-operator-controller-manager-695645577-b6l8r\" (UID: \"546b7cfd-e56a-413e-b157-8a1810969aec\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:55.983197 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.983170 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/546b7cfd-e56a-413e-b157-8a1810969aec-apiservice-cert\") pod \"opendatahub-operator-controller-manager-695645577-b6l8r\" (UID: \"546b7cfd-e56a-413e-b157-8a1810969aec\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:55.989156 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:55.989129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4zm4\" (UniqueName: \"kubernetes.io/projected/546b7cfd-e56a-413e-b157-8a1810969aec-kube-api-access-v4zm4\") pod \"opendatahub-operator-controller-manager-695645577-b6l8r\" (UID: \"546b7cfd-e56a-413e-b157-8a1810969aec\") " pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:56.003963 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:56.003924 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:32:56.148191 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:56.148164 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r"] Jul 23 00:32:56.150437 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:32:56.150409 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546b7cfd_e56a_413e_b157_8a1810969aec.slice/crio-9f7ec140288dc26fae61360554366491469947ed6969bf380f0214404457ec77 WatchSource:0}: Error finding container 9f7ec140288dc26fae61360554366491469947ed6969bf380f0214404457ec77: Status 404 returned error can't find the container with id 9f7ec140288dc26fae61360554366491469947ed6969bf380f0214404457ec77 Jul 23 00:32:56.287033 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:56.286943 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" event={"ID":"546b7cfd-e56a-413e-b157-8a1810969aec","Type":"ContainerStarted","Data":"9f7ec140288dc26fae61360554366491469947ed6969bf380f0214404457ec77"} Jul 23 00:32:56.382567 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:32:56.382496 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:00.316140 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:00.316036 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" event={"ID":"546b7cfd-e56a-413e-b157-8a1810969aec","Type":"ContainerStarted","Data":"859219795ad8f88b3ebd7553053ccf222fcabdcc6892afdbad22aaa2d4b91a79"} Jul 23 00:33:00.457249 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:00.457216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:00.553211 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:00.553173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:01.323595 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:01.323557 2564 generic.go:356] "Generic (PLEG): container finished" podID="546b7cfd-e56a-413e-b157-8a1810969aec" containerID="859219795ad8f88b3ebd7553053ccf222fcabdcc6892afdbad22aaa2d4b91a79" exitCode=0 Jul 23 00:33:01.324029 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:01.323650 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" event={"ID":"546b7cfd-e56a-413e-b157-8a1810969aec","Type":"ContainerDied","Data":"859219795ad8f88b3ebd7553053ccf222fcabdcc6892afdbad22aaa2d4b91a79"} Jul 23 00:33:02.330393 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:02.330350 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" event={"ID":"546b7cfd-e56a-413e-b157-8a1810969aec","Type":"ContainerStarted","Data":"759861e86403cca4e79346b7d09dff0d04b023c318637a862b315be5ba1307b6"} Jul 23 00:33:02.330841 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:02.330563 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:33:02.356068 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:02.356009 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" podStartSLOduration=1.9780912430000002 podStartE2EDuration="7.355989682s" totalImagesPullingTime="4.271503894s" totalInitContainerRuntime="1.106394545s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:32:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:33:02.3539544 +0000 UTC m=+574.849623138" watchObservedRunningTime="2026-07-23 00:33:02.355989682 +0000 UTC m=+574.851658395" Jul 23 00:33:09.072833 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:09.072785 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:09.131738 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:09.131701 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:09.523165 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:09.523125 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:09.584796 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:09.584746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:13.338194 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:13.338163 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-695645577-b6l8r" Jul 23 00:33:15.087058 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:15.087017 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:15.237664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:15.237626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:15.655310 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:15.655274 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:15.845486 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:15.845440 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-678df7485d-lxtmr"] Jul 23 00:33:15.856490 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:15.856463 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-678df7485d-lxtmr" Jul 23 00:33:15.861675 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:15.861644 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xpwtf\"" Jul 23 00:33:15.861921 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:15.861644 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 00:33:15.862031 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:15.861655 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 23 00:33:15.862109 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:15.861674 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 23 00:33:15.862185 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:15.861738 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 00:33:15.874315 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:15.874281 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-678df7485d-lxtmr"] Jul 23 00:33:15.903151 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:15.903101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:15.958397 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:15.958290 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2n2q\" (UniqueName: \"kubernetes.io/projected/c2241c6c-9470-4742-9352-0418c07a566b-kube-api-access-h2n2q\") pod \"kube-auth-proxy-678df7485d-lxtmr\" (UID: \"c2241c6c-9470-4742-9352-0418c07a566b\") " pod="openshift-ingress/kube-auth-proxy-678df7485d-lxtmr" Jul 23 00:33:15.958609 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:15.958439 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2241c6c-9470-4742-9352-0418c07a566b-tls-certs\") pod \"kube-auth-proxy-678df7485d-lxtmr\" (UID: \"c2241c6c-9470-4742-9352-0418c07a566b\") " pod="openshift-ingress/kube-auth-proxy-678df7485d-lxtmr" Jul 23 00:33:15.958609 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:15.958516 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2241c6c-9470-4742-9352-0418c07a566b-tmp\") pod \"kube-auth-proxy-678df7485d-lxtmr\" (UID: \"c2241c6c-9470-4742-9352-0418c07a566b\") " pod="openshift-ingress/kube-auth-proxy-678df7485d-lxtmr" Jul 23 00:33:16.059347 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:16.059304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2241c6c-9470-4742-9352-0418c07a566b-tmp\") pod \"kube-auth-proxy-678df7485d-lxtmr\" (UID: \"c2241c6c-9470-4742-9352-0418c07a566b\") " pod="openshift-ingress/kube-auth-proxy-678df7485d-lxtmr" Jul 23 00:33:16.059566 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:16.059367 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2n2q\" (UniqueName: \"kubernetes.io/projected/c2241c6c-9470-4742-9352-0418c07a566b-kube-api-access-h2n2q\") pod \"kube-auth-proxy-678df7485d-lxtmr\" (UID: \"c2241c6c-9470-4742-9352-0418c07a566b\") " pod="openshift-ingress/kube-auth-proxy-678df7485d-lxtmr" Jul 23 00:33:16.059566 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:16.059413 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2241c6c-9470-4742-9352-0418c07a566b-tls-certs\") pod \"kube-auth-proxy-678df7485d-lxtmr\" (UID: \"c2241c6c-9470-4742-9352-0418c07a566b\") " pod="openshift-ingress/kube-auth-proxy-678df7485d-lxtmr" Jul 23 00:33:16.061882 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:16.061848 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2241c6c-9470-4742-9352-0418c07a566b-tmp\") pod \"kube-auth-proxy-678df7485d-lxtmr\" (UID: \"c2241c6c-9470-4742-9352-0418c07a566b\") " pod="openshift-ingress/kube-auth-proxy-678df7485d-lxtmr" Jul 23 00:33:16.062016 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:16.061979 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2241c6c-9470-4742-9352-0418c07a566b-tls-certs\") pod \"kube-auth-proxy-678df7485d-lxtmr\" (UID: \"c2241c6c-9470-4742-9352-0418c07a566b\") " pod="openshift-ingress/kube-auth-proxy-678df7485d-lxtmr" Jul 23 00:33:16.069706 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:16.069672 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2n2q\" (UniqueName: \"kubernetes.io/projected/c2241c6c-9470-4742-9352-0418c07a566b-kube-api-access-h2n2q\") pod \"kube-auth-proxy-678df7485d-lxtmr\" (UID: \"c2241c6c-9470-4742-9352-0418c07a566b\") " pod="openshift-ingress/kube-auth-proxy-678df7485d-lxtmr" Jul 23 00:33:16.163832 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:16.163799 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:16.167664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:16.167633 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-678df7485d-lxtmr" Jul 23 00:33:16.237964 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:16.237801 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:16.335319 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:16.335293 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-678df7485d-lxtmr"] Jul 23 00:33:16.338250 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:33:16.338217 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2241c6c_9470_4742_9352_0418c07a566b.slice/crio-283a484182a23d0aee1706e6a3de40773d5d92894366d8d7faad82bdf88f55fa WatchSource:0}: Error finding container 283a484182a23d0aee1706e6a3de40773d5d92894366d8d7faad82bdf88f55fa: Status 404 returned error can't find the container with id 283a484182a23d0aee1706e6a3de40773d5d92894366d8d7faad82bdf88f55fa Jul 23 00:33:16.341313 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:16.341196 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:16.407774 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:16.407732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-678df7485d-lxtmr" event={"ID":"c2241c6c-9470-4742-9352-0418c07a566b","Type":"ContainerStarted","Data":"283a484182a23d0aee1706e6a3de40773d5d92894366d8d7faad82bdf88f55fa"} Jul 23 00:33:16.461043 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:16.461001 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:16.700075 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:16.700035 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:17.160472 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:17.160429 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:17.342875 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:17.342805 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:17.430551 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:17.430447 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:18.200519 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:18.200482 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:18.452364 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:18.452275 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:19.428432 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:19.428393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-678df7485d-lxtmr" event={"ID":"c2241c6c-9470-4742-9352-0418c07a566b","Type":"ContainerStarted","Data":"4d5df94e64e0a2cfc9c8a5a29caed36a4ab1043e49f6bb3bd2a52e361101da0e"} Jul 23 00:33:19.446052 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:19.445989 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-678df7485d-lxtmr" podStartSLOduration=1.82624828 podStartE2EDuration="4.445971415s" totalImagesPullingTime="2.619723135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:33:19.443757825 +0000 UTC m=+591.939426540" watchObservedRunningTime="2026-07-23 00:33:19.445971415 +0000 UTC m=+591.941640202" Jul 23 00:33:22.789183 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:22.789144 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:22.890897 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:22.890860 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:23.102565 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.102461 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg"] Jul 23 00:33:23.105797 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.105782 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.108606 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.108516 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 00:33:23.108717 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.108575 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-bc4xf\"" Jul 23 00:33:23.117083 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.117061 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg"] Jul 23 00:33:23.146241 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.146203 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:23.217446 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.217406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.217661 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.217454 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.217661 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.217483 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phpjg\" (UniqueName: \"kubernetes.io/projected/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-kube-api-access-phpjg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.217661 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.217508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.217661 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.217544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.217661 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.217573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.217661 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.217591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.217992 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.217737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.217992 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.217776 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.243137 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.243095 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:23.318460 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.318420 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.318460 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.318463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.318759 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.318485 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phpjg\" (UniqueName: \"kubernetes.io/projected/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-kube-api-access-phpjg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.318759 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.318629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.318759 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.318678 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.318759 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.318713 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.318759 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.318740 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.319063 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.318821 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.319063 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.318848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.319063 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.318910 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.319063 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.318971 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.319276 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.319187 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.319276 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.319271 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.319587 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.319568 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.321116 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.321095 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.321604 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.321584 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.327287 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.327264 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.327451 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.327430 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phpjg\" (UniqueName: \"kubernetes.io/projected/080c6928-b5fe-4a13-bba0-4ae0ef99aa7f-kube-api-access-phpjg\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4pxsjg\" (UID: \"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.418034 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.417987 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:23.431816 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.431777 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:23.622097 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:23.622000 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg"] Jul 23 00:33:23.624863 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:33:23.624827 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080c6928_b5fe_4a13_bba0_4ae0ef99aa7f.slice/crio-04513d07bd8b243eaa21368c9ed5a93a95e4a097fc38779c21c3a06e1e22a5cd WatchSource:0}: Error finding container 04513d07bd8b243eaa21368c9ed5a93a95e4a097fc38779c21c3a06e1e22a5cd: Status 404 returned error can't find the container with id 04513d07bd8b243eaa21368c9ed5a93a95e4a097fc38779c21c3a06e1e22a5cd Jul 23 00:33:24.457315 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:24.457236 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" event={"ID":"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f","Type":"ContainerStarted","Data":"04513d07bd8b243eaa21368c9ed5a93a95e4a097fc38779c21c3a06e1e22a5cd"} Jul 23 00:33:25.197105 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.197061 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm"] Jul 23 00:33:25.200941 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.200909 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:25.206632 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.206561 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 00:33:25.206632 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.206607 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 23 00:33:25.206974 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.206669 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 00:33:25.206974 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.206607 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 23 00:33:25.206974 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.206884 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 23 00:33:25.206974 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.206891 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-64hsz\"" Jul 23 00:33:25.222496 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.222447 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm"] Jul 23 00:33:25.338669 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.338630 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/77d9a36b-e5dd-478e-bdaa-aa4e1880b647-metrics-cert\") pod \"lws-controller-manager-85b95d447c-skrdm\" (UID: \"77d9a36b-e5dd-478e-bdaa-aa4e1880b647\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:25.338863 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.338694 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d54kk\" (UniqueName: \"kubernetes.io/projected/77d9a36b-e5dd-478e-bdaa-aa4e1880b647-kube-api-access-d54kk\") pod \"lws-controller-manager-85b95d447c-skrdm\" (UID: \"77d9a36b-e5dd-478e-bdaa-aa4e1880b647\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:25.338863 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.338737 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77d9a36b-e5dd-478e-bdaa-aa4e1880b647-cert\") pod \"lws-controller-manager-85b95d447c-skrdm\" (UID: \"77d9a36b-e5dd-478e-bdaa-aa4e1880b647\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:25.338863 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.338831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/77d9a36b-e5dd-478e-bdaa-aa4e1880b647-manager-config\") pod \"lws-controller-manager-85b95d447c-skrdm\" (UID: \"77d9a36b-e5dd-478e-bdaa-aa4e1880b647\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:25.440097 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.440049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/77d9a36b-e5dd-478e-bdaa-aa4e1880b647-manager-config\") pod \"lws-controller-manager-85b95d447c-skrdm\" (UID: \"77d9a36b-e5dd-478e-bdaa-aa4e1880b647\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:25.440296 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.440169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/77d9a36b-e5dd-478e-bdaa-aa4e1880b647-metrics-cert\") pod \"lws-controller-manager-85b95d447c-skrdm\" (UID: \"77d9a36b-e5dd-478e-bdaa-aa4e1880b647\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:25.440296 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.440221 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d54kk\" (UniqueName: \"kubernetes.io/projected/77d9a36b-e5dd-478e-bdaa-aa4e1880b647-kube-api-access-d54kk\") pod \"lws-controller-manager-85b95d447c-skrdm\" (UID: \"77d9a36b-e5dd-478e-bdaa-aa4e1880b647\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:25.440296 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.440266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77d9a36b-e5dd-478e-bdaa-aa4e1880b647-cert\") pod \"lws-controller-manager-85b95d447c-skrdm\" (UID: \"77d9a36b-e5dd-478e-bdaa-aa4e1880b647\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:25.440839 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.440807 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/77d9a36b-e5dd-478e-bdaa-aa4e1880b647-manager-config\") pod \"lws-controller-manager-85b95d447c-skrdm\" (UID: \"77d9a36b-e5dd-478e-bdaa-aa4e1880b647\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:25.443101 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.443077 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77d9a36b-e5dd-478e-bdaa-aa4e1880b647-cert\") pod \"lws-controller-manager-85b95d447c-skrdm\" (UID: \"77d9a36b-e5dd-478e-bdaa-aa4e1880b647\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:25.443214 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.443080 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/77d9a36b-e5dd-478e-bdaa-aa4e1880b647-metrics-cert\") pod \"lws-controller-manager-85b95d447c-skrdm\" (UID: \"77d9a36b-e5dd-478e-bdaa-aa4e1880b647\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:25.449273 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.449193 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d54kk\" (UniqueName: \"kubernetes.io/projected/77d9a36b-e5dd-478e-bdaa-aa4e1880b647-kube-api-access-d54kk\") pod \"lws-controller-manager-85b95d447c-skrdm\" (UID: \"77d9a36b-e5dd-478e-bdaa-aa4e1880b647\") " pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:25.513485 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:25.513447 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:26.424230 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:26.424177 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 00:33:26.424305 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:26.424275 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 00:33:26.424341 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:26.424305 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 00:33:26.554691 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:26.554654 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm"] Jul 23 00:33:26.560164 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:33:26.560127 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d9a36b_e5dd_478e_bdaa_aa4e1880b647.slice/crio-10d7468209c97d1a30af4d4cabdcfe3b8d647325ce8f08d59f45d1a6f4509d74 WatchSource:0}: Error finding container 10d7468209c97d1a30af4d4cabdcfe3b8d647325ce8f08d59f45d1a6f4509d74: Status 404 returned error can't find the container with id 10d7468209c97d1a30af4d4cabdcfe3b8d647325ce8f08d59f45d1a6f4509d74 Jul 23 00:33:26.932061 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:26.932018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:27.034769 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:27.034728 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:27.162057 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:27.162001 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:27.477324 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:27.477275 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" event={"ID":"080c6928-b5fe-4a13-bba0-4ae0ef99aa7f","Type":"ContainerStarted","Data":"b59f12a409865a1af928d8e88f181d4a99dac23f53ec3ea8ce9f2b9b50bfede6"} Jul 23 00:33:27.479086 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:27.479046 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" event={"ID":"77d9a36b-e5dd-478e-bdaa-aa4e1880b647","Type":"ContainerStarted","Data":"10d7468209c97d1a30af4d4cabdcfe3b8d647325ce8f08d59f45d1a6f4509d74"} Jul 23 00:33:27.493575 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:27.493478 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:27.510352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:27.510286 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" podStartSLOduration=1.713584913 podStartE2EDuration="4.510261395s" totalImagesPullingTime="2.796676482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:33:27.50781178 +0000 UTC m=+600.003480493" watchObservedRunningTime="2026-07-23 00:33:27.510261395 +0000 UTC m=+600.005930109" Jul 23 00:33:27.683905 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:27.683856 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:28.008423 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:28.008391 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:33:28.008655 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:28.008605 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:33:28.389728 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:28.389685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:28.418664 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:28.418623 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:28.423815 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:28.423785 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:28.486294 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:28.486253 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" event={"ID":"77d9a36b-e5dd-478e-bdaa-aa4e1880b647","Type":"ContainerStarted","Data":"032c4d19a23d9e05a07d1760c102435d668351d03d2ea7589e7f32d955e98f5f"} Jul 23 00:33:28.486520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:28.486500 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:28.486713 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:28.486694 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:28.487759 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:28.487737 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4pxsjg" Jul 23 00:33:28.515442 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:28.515387 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" podStartSLOduration=2.041268037 podStartE2EDuration="3.515369539s" totalImagesPullingTime="1.474101502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:33:28.513454746 +0000 UTC m=+601.009123460" watchObservedRunningTime="2026-07-23 00:33:28.515369539 +0000 UTC m=+601.011038252" Jul 23 00:33:28.533971 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:28.533932 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:29.325762 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:29.325723 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:29.429143 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:29.429109 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:31.988561 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:31.988453 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:32.048337 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:32.048292 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:39.495481 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:39.495442 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-85b95d447c-skrdm" Jul 23 00:33:49.411412 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:49.411366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:33:49.470017 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:33:49.469977 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:16.573832 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:16.573788 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:16.693463 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:16.693434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:16.792866 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:16.792829 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:17.465066 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:17.465031 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:17.668359 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:17.668323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:32.080730 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.080614 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r"] Jul 23 00:34:32.085663 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.084334 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" Jul 23 00:34:32.087782 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.087757 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 00:34:32.089153 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.089116 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 00:34:32.089293 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.089226 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nzwvk\"" Jul 23 00:34:32.092198 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.092167 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r"] Jul 23 00:34:32.190928 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.190866 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hns24\" (UniqueName: \"kubernetes.io/projected/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-kube-api-access-hns24\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r\" (UID: \"87859bfc-58b8-4234-acc0-bbc1e5e1baa2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" Jul 23 00:34:32.191142 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.190989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r\" (UID: \"87859bfc-58b8-4234-acc0-bbc1e5e1baa2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" Jul 23 00:34:32.191142 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.191077 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r\" (UID: \"87859bfc-58b8-4234-acc0-bbc1e5e1baa2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" Jul 23 00:34:32.291852 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.291804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r\" (UID: \"87859bfc-58b8-4234-acc0-bbc1e5e1baa2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" Jul 23 00:34:32.292008 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.291892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r\" (UID: \"87859bfc-58b8-4234-acc0-bbc1e5e1baa2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" Jul 23 00:34:32.292008 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.291916 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hns24\" (UniqueName: \"kubernetes.io/projected/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-kube-api-access-hns24\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r\" (UID: \"87859bfc-58b8-4234-acc0-bbc1e5e1baa2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" Jul 23 00:34:32.292214 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.292192 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r\" (UID: \"87859bfc-58b8-4234-acc0-bbc1e5e1baa2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" Jul 23 00:34:32.292254 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.292221 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r\" (UID: \"87859bfc-58b8-4234-acc0-bbc1e5e1baa2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" Jul 23 00:34:32.301166 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.301129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hns24\" (UniqueName: \"kubernetes.io/projected/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-kube-api-access-hns24\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r\" (UID: \"87859bfc-58b8-4234-acc0-bbc1e5e1baa2\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" Jul 23 00:34:32.398617 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.398581 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" Jul 23 00:34:32.539794 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.539769 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r"] Jul 23 00:34:32.542620 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:34:32.542591 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87859bfc_58b8_4234_acc0_bbc1e5e1baa2.slice/crio-fb3844a308da46b8446cae88f50ef1ca2bbbd184384cb7ce32afd109759d1cf9 WatchSource:0}: Error finding container fb3844a308da46b8446cae88f50ef1ca2bbbd184384cb7ce32afd109759d1cf9: Status 404 returned error can't find the container with id fb3844a308da46b8446cae88f50ef1ca2bbbd184384cb7ce32afd109759d1cf9 Jul 23 00:34:32.676110 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.676026 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm"] Jul 23 00:34:32.678743 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.678724 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" Jul 23 00:34:32.687157 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.687131 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm"] Jul 23 00:34:32.797599 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.797521 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e83cf781-638d-4728-bc84-7e3dc7fb8089-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm\" (UID: \"e83cf781-638d-4728-bc84-7e3dc7fb8089\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" Jul 23 00:34:32.797773 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.797708 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k22lm\" (UniqueName: \"kubernetes.io/projected/e83cf781-638d-4728-bc84-7e3dc7fb8089-kube-api-access-k22lm\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm\" (UID: \"e83cf781-638d-4728-bc84-7e3dc7fb8089\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" Jul 23 00:34:32.797773 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.797758 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e83cf781-638d-4728-bc84-7e3dc7fb8089-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm\" (UID: \"e83cf781-638d-4728-bc84-7e3dc7fb8089\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" Jul 23 00:34:32.825806 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.825773 2564 generic.go:356] "Generic (PLEG): container finished" podID="87859bfc-58b8-4234-acc0-bbc1e5e1baa2" containerID="ad8991ebba5b79d9d7d3e89dd6f669e3c6cc5e8751f2186705151cb316823274" exitCode=0 Jul 23 00:34:32.825806 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.825809 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" event={"ID":"87859bfc-58b8-4234-acc0-bbc1e5e1baa2","Type":"ContainerDied","Data":"ad8991ebba5b79d9d7d3e89dd6f669e3c6cc5e8751f2186705151cb316823274"} Jul 23 00:34:32.826147 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.825856 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" event={"ID":"87859bfc-58b8-4234-acc0-bbc1e5e1baa2","Type":"ContainerStarted","Data":"fb3844a308da46b8446cae88f50ef1ca2bbbd184384cb7ce32afd109759d1cf9"} Jul 23 00:34:32.826337 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.826320 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:34:32.899065 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.899022 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22lm\" (UniqueName: \"kubernetes.io/projected/e83cf781-638d-4728-bc84-7e3dc7fb8089-kube-api-access-k22lm\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm\" (UID: \"e83cf781-638d-4728-bc84-7e3dc7fb8089\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" Jul 23 00:34:32.899065 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.899085 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e83cf781-638d-4728-bc84-7e3dc7fb8089-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm\" (UID: \"e83cf781-638d-4728-bc84-7e3dc7fb8089\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" Jul 23 00:34:32.899339 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.899107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e83cf781-638d-4728-bc84-7e3dc7fb8089-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm\" (UID: \"e83cf781-638d-4728-bc84-7e3dc7fb8089\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" Jul 23 00:34:32.899486 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.899469 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e83cf781-638d-4728-bc84-7e3dc7fb8089-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm\" (UID: \"e83cf781-638d-4728-bc84-7e3dc7fb8089\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" Jul 23 00:34:32.899583 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.899521 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e83cf781-638d-4728-bc84-7e3dc7fb8089-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm\" (UID: \"e83cf781-638d-4728-bc84-7e3dc7fb8089\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" Jul 23 00:34:32.908299 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:32.908267 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k22lm\" (UniqueName: \"kubernetes.io/projected/e83cf781-638d-4728-bc84-7e3dc7fb8089-kube-api-access-k22lm\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm\" (UID: \"e83cf781-638d-4728-bc84-7e3dc7fb8089\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" Jul 23 00:34:33.007555 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.007424 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" Jul 23 00:34:33.075915 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.075885 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz"] Jul 23 00:34:33.078826 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.078804 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" Jul 23 00:34:33.087679 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.087649 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz"] Jul 23 00:34:33.148005 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.147976 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm"] Jul 23 00:34:33.149635 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:34:33.149610 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83cf781_638d_4728_bc84_7e3dc7fb8089.slice/crio-90a797f788937aeff3a707376dc1fff4b9119c8c3bfe346517da5011e4f981ab WatchSource:0}: Error finding container 90a797f788937aeff3a707376dc1fff4b9119c8c3bfe346517da5011e4f981ab: Status 404 returned error can't find the container with id 90a797f788937aeff3a707376dc1fff4b9119c8c3bfe346517da5011e4f981ab Jul 23 00:34:33.202401 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.202054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz\" (UID: \"02a19ed9-900d-4de6-89e6-ffcd229a4b9a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" Jul 23 00:34:33.202401 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.202184 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5wx2\" (UniqueName: \"kubernetes.io/projected/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-kube-api-access-j5wx2\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz\" (UID: \"02a19ed9-900d-4de6-89e6-ffcd229a4b9a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" Jul 23 00:34:33.202401 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.202253 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz\" (UID: \"02a19ed9-900d-4de6-89e6-ffcd229a4b9a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" Jul 23 00:34:33.303783 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.303747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz\" (UID: \"02a19ed9-900d-4de6-89e6-ffcd229a4b9a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" Jul 23 00:34:33.303963 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.303825 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5wx2\" (UniqueName: \"kubernetes.io/projected/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-kube-api-access-j5wx2\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz\" (UID: \"02a19ed9-900d-4de6-89e6-ffcd229a4b9a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" Jul 23 00:34:33.303963 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.303922 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz\" (UID: \"02a19ed9-900d-4de6-89e6-ffcd229a4b9a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" Jul 23 00:34:33.304244 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.304219 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz\" (UID: \"02a19ed9-900d-4de6-89e6-ffcd229a4b9a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" Jul 23 00:34:33.304281 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.304244 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz\" (UID: \"02a19ed9-900d-4de6-89e6-ffcd229a4b9a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" Jul 23 00:34:33.313138 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.313110 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5wx2\" (UniqueName: \"kubernetes.io/projected/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-kube-api-access-j5wx2\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz\" (UID: \"02a19ed9-900d-4de6-89e6-ffcd229a4b9a\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" Jul 23 00:34:33.390647 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.390609 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" Jul 23 00:34:33.488599 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.488560 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv"] Jul 23 00:34:33.491592 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.491563 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" Jul 23 00:34:33.502402 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.502354 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv"] Jul 23 00:34:33.606787 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.606753 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3c100f9-9818-4a56-80a3-77544ae18cf1-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv\" (UID: \"b3c100f9-9818-4a56-80a3-77544ae18cf1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" Jul 23 00:34:33.606928 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.606878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3c100f9-9818-4a56-80a3-77544ae18cf1-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv\" (UID: \"b3c100f9-9818-4a56-80a3-77544ae18cf1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" Jul 23 00:34:33.606991 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.606924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9gq6\" (UniqueName: \"kubernetes.io/projected/b3c100f9-9818-4a56-80a3-77544ae18cf1-kube-api-access-d9gq6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv\" (UID: \"b3c100f9-9818-4a56-80a3-77544ae18cf1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" Jul 23 00:34:33.656246 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.656220 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz"] Jul 23 00:34:33.686829 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:34:33.686787 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a19ed9_900d_4de6_89e6_ffcd229a4b9a.slice/crio-9aa373dd16308c6550c1a2211e8c3b76e174ec5d7dc098bdc5b6b1e86ffeb2f3 WatchSource:0}: Error finding container 9aa373dd16308c6550c1a2211e8c3b76e174ec5d7dc098bdc5b6b1e86ffeb2f3: Status 404 returned error can't find the container with id 9aa373dd16308c6550c1a2211e8c3b76e174ec5d7dc098bdc5b6b1e86ffeb2f3 Jul 23 00:34:33.707408 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.707380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3c100f9-9818-4a56-80a3-77544ae18cf1-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv\" (UID: \"b3c100f9-9818-4a56-80a3-77544ae18cf1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" Jul 23 00:34:33.707547 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.707419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9gq6\" (UniqueName: \"kubernetes.io/projected/b3c100f9-9818-4a56-80a3-77544ae18cf1-kube-api-access-d9gq6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv\" (UID: \"b3c100f9-9818-4a56-80a3-77544ae18cf1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" Jul 23 00:34:33.707547 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.707486 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3c100f9-9818-4a56-80a3-77544ae18cf1-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv\" (UID: \"b3c100f9-9818-4a56-80a3-77544ae18cf1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" Jul 23 00:34:33.707754 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.707735 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3c100f9-9818-4a56-80a3-77544ae18cf1-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv\" (UID: \"b3c100f9-9818-4a56-80a3-77544ae18cf1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" Jul 23 00:34:33.707806 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.707790 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3c100f9-9818-4a56-80a3-77544ae18cf1-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv\" (UID: \"b3c100f9-9818-4a56-80a3-77544ae18cf1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" Jul 23 00:34:33.716048 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.716022 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9gq6\" (UniqueName: \"kubernetes.io/projected/b3c100f9-9818-4a56-80a3-77544ae18cf1-kube-api-access-d9gq6\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv\" (UID: \"b3c100f9-9818-4a56-80a3-77544ae18cf1\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" Jul 23 00:34:33.804104 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.804060 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" Jul 23 00:34:33.832982 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.832947 2564 generic.go:356] "Generic (PLEG): container finished" podID="87859bfc-58b8-4234-acc0-bbc1e5e1baa2" containerID="23a9287aefc672544c889bcf837c87d99a7984e57e91ce78f31a12728f2681b3" exitCode=0 Jul 23 00:34:33.833191 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.833039 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" event={"ID":"87859bfc-58b8-4234-acc0-bbc1e5e1baa2","Type":"ContainerDied","Data":"23a9287aefc672544c889bcf837c87d99a7984e57e91ce78f31a12728f2681b3"} Jul 23 00:34:33.834869 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.834843 2564 generic.go:356] "Generic (PLEG): container finished" podID="e83cf781-638d-4728-bc84-7e3dc7fb8089" containerID="da8bd087a51db15f074a4b10c1fef18c73c2306dd97e9503dc88cbab4fab0cac" exitCode=0 Jul 23 00:34:33.834961 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.834923 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" event={"ID":"e83cf781-638d-4728-bc84-7e3dc7fb8089","Type":"ContainerDied","Data":"da8bd087a51db15f074a4b10c1fef18c73c2306dd97e9503dc88cbab4fab0cac"} Jul 23 00:34:33.835016 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.834983 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" event={"ID":"e83cf781-638d-4728-bc84-7e3dc7fb8089","Type":"ContainerStarted","Data":"90a797f788937aeff3a707376dc1fff4b9119c8c3bfe346517da5011e4f981ab"} Jul 23 00:34:33.836874 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.836849 2564 generic.go:356] "Generic (PLEG): container finished" podID="02a19ed9-900d-4de6-89e6-ffcd229a4b9a" containerID="6401d309c57864a143df15a6f1cea2b2d64703c90a1ed8750a2efcd944a473dc" exitCode=0 Jul 23 00:34:33.837001 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.836872 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" event={"ID":"02a19ed9-900d-4de6-89e6-ffcd229a4b9a","Type":"ContainerDied","Data":"6401d309c57864a143df15a6f1cea2b2d64703c90a1ed8750a2efcd944a473dc"} Jul 23 00:34:33.837001 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.836927 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" event={"ID":"02a19ed9-900d-4de6-89e6-ffcd229a4b9a","Type":"ContainerStarted","Data":"9aa373dd16308c6550c1a2211e8c3b76e174ec5d7dc098bdc5b6b1e86ffeb2f3"} Jul 23 00:34:33.961245 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:33.961220 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv"] Jul 23 00:34:33.964497 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:34:33.964465 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c100f9_9818_4a56_80a3_77544ae18cf1.slice/crio-f1c102af8e0caabbca557ea3041677c2ecee23d2cc468c6456105d6b0627f918 WatchSource:0}: Error finding container f1c102af8e0caabbca557ea3041677c2ecee23d2cc468c6456105d6b0627f918: Status 404 returned error can't find the container with id f1c102af8e0caabbca557ea3041677c2ecee23d2cc468c6456105d6b0627f918 Jul 23 00:34:34.851786 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:34.851750 2564 generic.go:356] "Generic (PLEG): container finished" podID="02a19ed9-900d-4de6-89e6-ffcd229a4b9a" containerID="4a21b556223be0aa83ce1c11e07037ad0342159349f5d562453835d9f30d9919" exitCode=0 Jul 23 00:34:34.852214 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:34.851837 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" event={"ID":"02a19ed9-900d-4de6-89e6-ffcd229a4b9a","Type":"ContainerDied","Data":"4a21b556223be0aa83ce1c11e07037ad0342159349f5d562453835d9f30d9919"} Jul 23 00:34:34.854277 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:34.854251 2564 generic.go:356] "Generic (PLEG): container finished" podID="87859bfc-58b8-4234-acc0-bbc1e5e1baa2" containerID="32bbc7ab073740873ebde7c0b44ffb714838ec248223d1673519f0a3fe660bec" exitCode=0 Jul 23 00:34:34.854367 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:34.854340 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" event={"ID":"87859bfc-58b8-4234-acc0-bbc1e5e1baa2","Type":"ContainerDied","Data":"32bbc7ab073740873ebde7c0b44ffb714838ec248223d1673519f0a3fe660bec"} Jul 23 00:34:34.856186 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:34.856163 2564 generic.go:356] "Generic (PLEG): container finished" podID="e83cf781-638d-4728-bc84-7e3dc7fb8089" containerID="3c2d7c5e6874a398027ce70e75b6468fec23196b0692a6eba0622359171da6df" exitCode=0 Jul 23 00:34:34.856311 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:34.856246 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" event={"ID":"e83cf781-638d-4728-bc84-7e3dc7fb8089","Type":"ContainerDied","Data":"3c2d7c5e6874a398027ce70e75b6468fec23196b0692a6eba0622359171da6df"} Jul 23 00:34:34.857965 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:34.857895 2564 generic.go:356] "Generic (PLEG): container finished" podID="b3c100f9-9818-4a56-80a3-77544ae18cf1" containerID="6118a3b76a51a3f7ca4e458e4ebc88e640d3b84543c65c38fff27b663d825138" exitCode=0 Jul 23 00:34:34.857965 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:34.857943 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" event={"ID":"b3c100f9-9818-4a56-80a3-77544ae18cf1","Type":"ContainerDied","Data":"6118a3b76a51a3f7ca4e458e4ebc88e640d3b84543c65c38fff27b663d825138"} Jul 23 00:34:34.858111 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:34.857976 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" event={"ID":"b3c100f9-9818-4a56-80a3-77544ae18cf1","Type":"ContainerStarted","Data":"f1c102af8e0caabbca557ea3041677c2ecee23d2cc468c6456105d6b0627f918"} Jul 23 00:34:35.868150 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:35.868113 2564 generic.go:356] "Generic (PLEG): container finished" podID="02a19ed9-900d-4de6-89e6-ffcd229a4b9a" containerID="b1f328bac433e7ec8b80ba806a565d8bf4720ca939b01effd6df0d5f3084eb48" exitCode=0 Jul 23 00:34:35.868603 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:35.868191 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" event={"ID":"02a19ed9-900d-4de6-89e6-ffcd229a4b9a","Type":"ContainerDied","Data":"b1f328bac433e7ec8b80ba806a565d8bf4720ca939b01effd6df0d5f3084eb48"} Jul 23 00:34:35.870365 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:35.870342 2564 generic.go:356] "Generic (PLEG): container finished" podID="e83cf781-638d-4728-bc84-7e3dc7fb8089" containerID="dcee34484be3a346e757faa474f3ca1cf13563ed9560ffa738b6e8c4fa7b7714" exitCode=0 Jul 23 00:34:35.870496 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:35.870409 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" event={"ID":"e83cf781-638d-4728-bc84-7e3dc7fb8089","Type":"ContainerDied","Data":"dcee34484be3a346e757faa474f3ca1cf13563ed9560ffa738b6e8c4fa7b7714"} Jul 23 00:34:35.872209 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:35.872187 2564 generic.go:356] "Generic (PLEG): container finished" podID="b3c100f9-9818-4a56-80a3-77544ae18cf1" containerID="b062bcac89938952cc9ca2726773b487c575a9823078e8bdc177f6cdb83328da" exitCode=0 Jul 23 00:34:35.872334 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:35.872269 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" event={"ID":"b3c100f9-9818-4a56-80a3-77544ae18cf1","Type":"ContainerDied","Data":"b062bcac89938952cc9ca2726773b487c575a9823078e8bdc177f6cdb83328da"} Jul 23 00:34:36.009630 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:36.009598 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" Jul 23 00:34:36.133903 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:36.133867 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-util\") pod \"87859bfc-58b8-4234-acc0-bbc1e5e1baa2\" (UID: \"87859bfc-58b8-4234-acc0-bbc1e5e1baa2\") " Jul 23 00:34:36.134101 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:36.133970 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-bundle\") pod \"87859bfc-58b8-4234-acc0-bbc1e5e1baa2\" (UID: \"87859bfc-58b8-4234-acc0-bbc1e5e1baa2\") " Jul 23 00:34:36.134101 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:36.134016 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hns24\" (UniqueName: \"kubernetes.io/projected/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-kube-api-access-hns24\") pod \"87859bfc-58b8-4234-acc0-bbc1e5e1baa2\" (UID: \"87859bfc-58b8-4234-acc0-bbc1e5e1baa2\") " Jul 23 00:34:36.134595 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:36.134558 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-bundle" (OuterVolumeSpecName: "bundle") pod "87859bfc-58b8-4234-acc0-bbc1e5e1baa2" (UID: "87859bfc-58b8-4234-acc0-bbc1e5e1baa2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:34:36.136352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:36.136300 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-kube-api-access-hns24" (OuterVolumeSpecName: "kube-api-access-hns24") pod "87859bfc-58b8-4234-acc0-bbc1e5e1baa2" (UID: "87859bfc-58b8-4234-acc0-bbc1e5e1baa2"). InnerVolumeSpecName "kube-api-access-hns24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:34:36.139889 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:36.139851 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-util" (OuterVolumeSpecName: "util") pod "87859bfc-58b8-4234-acc0-bbc1e5e1baa2" (UID: "87859bfc-58b8-4234-acc0-bbc1e5e1baa2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:34:36.234931 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:36.234887 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-bundle\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:34:36.234931 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:36.234926 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hns24\" (UniqueName: \"kubernetes.io/projected/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-kube-api-access-hns24\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:34:36.234931 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:36.234939 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/87859bfc-58b8-4234-acc0-bbc1e5e1baa2-util\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:34:36.879892 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:36.879857 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" event={"ID":"87859bfc-58b8-4234-acc0-bbc1e5e1baa2","Type":"ContainerDied","Data":"fb3844a308da46b8446cae88f50ef1ca2bbbd184384cb7ce32afd109759d1cf9"} Jul 23 00:34:36.880328 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:36.879910 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb3844a308da46b8446cae88f50ef1ca2bbbd184384cb7ce32afd109759d1cf9" Jul 23 00:34:36.880328 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:36.879870 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r" Jul 23 00:34:36.882045 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:36.882017 2564 generic.go:356] "Generic (PLEG): container finished" podID="b3c100f9-9818-4a56-80a3-77544ae18cf1" containerID="3b4dd6bfed79e64a38fd3af6136b73dbaf805b8a0808e845f6f7b39bd363c1ba" exitCode=0 Jul 23 00:34:36.882176 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:36.882100 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" event={"ID":"b3c100f9-9818-4a56-80a3-77544ae18cf1","Type":"ContainerDied","Data":"3b4dd6bfed79e64a38fd3af6136b73dbaf805b8a0808e845f6f7b39bd363c1ba"} Jul 23 00:34:37.038098 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.038071 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" Jul 23 00:34:37.040248 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.040227 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" Jul 23 00:34:37.143648 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.143610 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e83cf781-638d-4728-bc84-7e3dc7fb8089-util\") pod \"e83cf781-638d-4728-bc84-7e3dc7fb8089\" (UID: \"e83cf781-638d-4728-bc84-7e3dc7fb8089\") " Jul 23 00:34:37.143812 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.143661 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e83cf781-638d-4728-bc84-7e3dc7fb8089-bundle\") pod \"e83cf781-638d-4728-bc84-7e3dc7fb8089\" (UID: \"e83cf781-638d-4728-bc84-7e3dc7fb8089\") " Jul 23 00:34:37.143812 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.143708 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k22lm\" (UniqueName: \"kubernetes.io/projected/e83cf781-638d-4728-bc84-7e3dc7fb8089-kube-api-access-k22lm\") pod \"e83cf781-638d-4728-bc84-7e3dc7fb8089\" (UID: \"e83cf781-638d-4728-bc84-7e3dc7fb8089\") " Jul 23 00:34:37.143812 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.143739 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-util\") pod \"02a19ed9-900d-4de6-89e6-ffcd229a4b9a\" (UID: \"02a19ed9-900d-4de6-89e6-ffcd229a4b9a\") " Jul 23 00:34:37.143812 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.143797 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-bundle\") pod \"02a19ed9-900d-4de6-89e6-ffcd229a4b9a\" (UID: \"02a19ed9-900d-4de6-89e6-ffcd229a4b9a\") " Jul 23 00:34:37.143812 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.143813 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5wx2\" (UniqueName: \"kubernetes.io/projected/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-kube-api-access-j5wx2\") pod \"02a19ed9-900d-4de6-89e6-ffcd229a4b9a\" (UID: \"02a19ed9-900d-4de6-89e6-ffcd229a4b9a\") " Jul 23 00:34:37.144410 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.144296 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83cf781-638d-4728-bc84-7e3dc7fb8089-bundle" (OuterVolumeSpecName: "bundle") pod "e83cf781-638d-4728-bc84-7e3dc7fb8089" (UID: "e83cf781-638d-4728-bc84-7e3dc7fb8089"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:34:37.144410 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.144363 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-bundle" (OuterVolumeSpecName: "bundle") pod "02a19ed9-900d-4de6-89e6-ffcd229a4b9a" (UID: "02a19ed9-900d-4de6-89e6-ffcd229a4b9a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:34:37.146033 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.146005 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83cf781-638d-4728-bc84-7e3dc7fb8089-kube-api-access-k22lm" (OuterVolumeSpecName: "kube-api-access-k22lm") pod "e83cf781-638d-4728-bc84-7e3dc7fb8089" (UID: "e83cf781-638d-4728-bc84-7e3dc7fb8089"). InnerVolumeSpecName "kube-api-access-k22lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:34:37.146147 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.146103 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-kube-api-access-j5wx2" (OuterVolumeSpecName: "kube-api-access-j5wx2") pod "02a19ed9-900d-4de6-89e6-ffcd229a4b9a" (UID: "02a19ed9-900d-4de6-89e6-ffcd229a4b9a"). InnerVolumeSpecName "kube-api-access-j5wx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:34:37.153001 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.152970 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e83cf781-638d-4728-bc84-7e3dc7fb8089-util" (OuterVolumeSpecName: "util") pod "e83cf781-638d-4728-bc84-7e3dc7fb8089" (UID: "e83cf781-638d-4728-bc84-7e3dc7fb8089"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:34:37.153084 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.153055 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-util" (OuterVolumeSpecName: "util") pod "02a19ed9-900d-4de6-89e6-ffcd229a4b9a" (UID: "02a19ed9-900d-4de6-89e6-ffcd229a4b9a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:34:37.244756 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.244713 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e83cf781-638d-4728-bc84-7e3dc7fb8089-util\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:34:37.244756 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.244748 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e83cf781-638d-4728-bc84-7e3dc7fb8089-bundle\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:34:37.244756 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.244760 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k22lm\" (UniqueName: \"kubernetes.io/projected/e83cf781-638d-4728-bc84-7e3dc7fb8089-kube-api-access-k22lm\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:34:37.244991 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.244771 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-util\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:34:37.244991 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.244779 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-bundle\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:34:37.244991 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.244787 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5wx2\" (UniqueName: \"kubernetes.io/projected/02a19ed9-900d-4de6-89e6-ffcd229a4b9a-kube-api-access-j5wx2\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:34:37.892091 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.892052 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" Jul 23 00:34:37.892567 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.892052 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm" event={"ID":"e83cf781-638d-4728-bc84-7e3dc7fb8089","Type":"ContainerDied","Data":"90a797f788937aeff3a707376dc1fff4b9119c8c3bfe346517da5011e4f981ab"} Jul 23 00:34:37.892567 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.892255 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a797f788937aeff3a707376dc1fff4b9119c8c3bfe346517da5011e4f981ab" Jul 23 00:34:37.904933 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.904897 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" Jul 23 00:34:37.905843 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.905008 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz" event={"ID":"02a19ed9-900d-4de6-89e6-ffcd229a4b9a","Type":"ContainerDied","Data":"9aa373dd16308c6550c1a2211e8c3b76e174ec5d7dc098bdc5b6b1e86ffeb2f3"} Jul 23 00:34:37.905843 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:37.905111 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9aa373dd16308c6550c1a2211e8c3b76e174ec5d7dc098bdc5b6b1e86ffeb2f3" Jul 23 00:34:38.043386 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:38.043359 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" Jul 23 00:34:38.153050 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:38.152947 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3c100f9-9818-4a56-80a3-77544ae18cf1-util\") pod \"b3c100f9-9818-4a56-80a3-77544ae18cf1\" (UID: \"b3c100f9-9818-4a56-80a3-77544ae18cf1\") " Jul 23 00:34:38.153050 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:38.153007 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9gq6\" (UniqueName: \"kubernetes.io/projected/b3c100f9-9818-4a56-80a3-77544ae18cf1-kube-api-access-d9gq6\") pod \"b3c100f9-9818-4a56-80a3-77544ae18cf1\" (UID: \"b3c100f9-9818-4a56-80a3-77544ae18cf1\") " Jul 23 00:34:38.153282 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:38.153111 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3c100f9-9818-4a56-80a3-77544ae18cf1-bundle\") pod \"b3c100f9-9818-4a56-80a3-77544ae18cf1\" (UID: \"b3c100f9-9818-4a56-80a3-77544ae18cf1\") " Jul 23 00:34:38.153764 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:38.153718 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c100f9-9818-4a56-80a3-77544ae18cf1-bundle" (OuterVolumeSpecName: "bundle") pod "b3c100f9-9818-4a56-80a3-77544ae18cf1" (UID: "b3c100f9-9818-4a56-80a3-77544ae18cf1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:34:38.155218 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:38.155189 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c100f9-9818-4a56-80a3-77544ae18cf1-kube-api-access-d9gq6" (OuterVolumeSpecName: "kube-api-access-d9gq6") pod "b3c100f9-9818-4a56-80a3-77544ae18cf1" (UID: "b3c100f9-9818-4a56-80a3-77544ae18cf1"). InnerVolumeSpecName "kube-api-access-d9gq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:34:38.160512 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:38.160456 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3c100f9-9818-4a56-80a3-77544ae18cf1-util" (OuterVolumeSpecName: "util") pod "b3c100f9-9818-4a56-80a3-77544ae18cf1" (UID: "b3c100f9-9818-4a56-80a3-77544ae18cf1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 00:34:38.254851 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:38.254807 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9gq6\" (UniqueName: \"kubernetes.io/projected/b3c100f9-9818-4a56-80a3-77544ae18cf1-kube-api-access-d9gq6\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:34:38.254851 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:38.254845 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3c100f9-9818-4a56-80a3-77544ae18cf1-bundle\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:34:38.254851 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:38.254856 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3c100f9-9818-4a56-80a3-77544ae18cf1-util\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:34:38.912900 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:38.912866 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" Jul 23 00:34:38.913272 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:38.912900 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv" event={"ID":"b3c100f9-9818-4a56-80a3-77544ae18cf1","Type":"ContainerDied","Data":"f1c102af8e0caabbca557ea3041677c2ecee23d2cc468c6456105d6b0627f918"} Jul 23 00:34:38.913272 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:38.912949 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1c102af8e0caabbca557ea3041677c2ecee23d2cc468c6456105d6b0627f918" Jul 23 00:34:43.219768 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.219731 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b756d7cd5-fdlsn"] Jul 23 00:34:43.220251 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220230 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3c100f9-9818-4a56-80a3-77544ae18cf1" containerName="pull" Jul 23 00:34:43.220301 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220255 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3c100f9-9818-4a56-80a3-77544ae18cf1" containerName="pull" Jul 23 00:34:43.220301 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220283 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83cf781-638d-4728-bc84-7e3dc7fb8089" containerName="util" Jul 23 00:34:43.220301 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220290 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83cf781-638d-4728-bc84-7e3dc7fb8089" containerName="util" Jul 23 00:34:43.220301 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220297 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3c100f9-9818-4a56-80a3-77544ae18cf1" containerName="extract" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220304 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3c100f9-9818-4a56-80a3-77544ae18cf1" containerName="extract" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220315 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83cf781-638d-4728-bc84-7e3dc7fb8089" containerName="pull" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220324 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83cf781-638d-4728-bc84-7e3dc7fb8089" containerName="pull" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220334 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87859bfc-58b8-4234-acc0-bbc1e5e1baa2" containerName="pull" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220339 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="87859bfc-58b8-4234-acc0-bbc1e5e1baa2" containerName="pull" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220346 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a19ed9-900d-4de6-89e6-ffcd229a4b9a" containerName="util" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220351 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a19ed9-900d-4de6-89e6-ffcd229a4b9a" containerName="util" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220370 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87859bfc-58b8-4234-acc0-bbc1e5e1baa2" containerName="extract" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220375 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="87859bfc-58b8-4234-acc0-bbc1e5e1baa2" containerName="extract" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220389 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a19ed9-900d-4de6-89e6-ffcd229a4b9a" containerName="pull" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220394 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a19ed9-900d-4de6-89e6-ffcd229a4b9a" containerName="pull" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220399 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e83cf781-638d-4728-bc84-7e3dc7fb8089" containerName="extract" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220404 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e83cf781-638d-4728-bc84-7e3dc7fb8089" containerName="extract" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220414 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3c100f9-9818-4a56-80a3-77544ae18cf1" containerName="util" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220419 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3c100f9-9818-4a56-80a3-77544ae18cf1" containerName="util" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220426 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87859bfc-58b8-4234-acc0-bbc1e5e1baa2" containerName="util" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220431 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="87859bfc-58b8-4234-acc0-bbc1e5e1baa2" containerName="util" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220438 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a19ed9-900d-4de6-89e6-ffcd229a4b9a" containerName="extract" Jul 23 00:34:43.220437 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220443 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a19ed9-900d-4de6-89e6-ffcd229a4b9a" containerName="extract" Jul 23 00:34:43.221012 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220504 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e83cf781-638d-4728-bc84-7e3dc7fb8089" containerName="extract" Jul 23 00:34:43.221012 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220512 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3c100f9-9818-4a56-80a3-77544ae18cf1" containerName="extract" Jul 23 00:34:43.221012 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220521 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="87859bfc-58b8-4234-acc0-bbc1e5e1baa2" containerName="extract" Jul 23 00:34:43.221012 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.220541 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="02a19ed9-900d-4de6-89e6-ffcd229a4b9a" containerName="extract" Jul 23 00:34:43.222874 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.222857 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.234095 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.234063 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b756d7cd5-fdlsn"] Jul 23 00:34:43.274188 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.274144 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:43.305373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.305333 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2f00827-2629-4285-8b64-da272361f851-console-serving-cert\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.305373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.305374 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2f00827-2629-4285-8b64-da272361f851-console-oauth-config\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.305640 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.305395 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2f00827-2629-4285-8b64-da272361f851-trusted-ca-bundle\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.305640 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.305453 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f00827-2629-4285-8b64-da272361f851-service-ca\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.305640 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.305473 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq8qz\" (UniqueName: \"kubernetes.io/projected/b2f00827-2629-4285-8b64-da272361f851-kube-api-access-cq8qz\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.305640 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.305511 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2f00827-2629-4285-8b64-da272361f851-console-config\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.305640 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.305585 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2f00827-2629-4285-8b64-da272361f851-oauth-serving-cert\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.328558 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.328497 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:43.406918 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.406879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2f00827-2629-4285-8b64-da272361f851-console-serving-cert\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.406918 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.406919 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2f00827-2629-4285-8b64-da272361f851-console-oauth-config\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.407156 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.406939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2f00827-2629-4285-8b64-da272361f851-trusted-ca-bundle\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.407156 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.406963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f00827-2629-4285-8b64-da272361f851-service-ca\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.407156 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.406996 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq8qz\" (UniqueName: \"kubernetes.io/projected/b2f00827-2629-4285-8b64-da272361f851-kube-api-access-cq8qz\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.407156 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.407049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2f00827-2629-4285-8b64-da272361f851-console-config\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.407156 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.407096 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2f00827-2629-4285-8b64-da272361f851-oauth-serving-cert\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.408018 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.407989 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f00827-2629-4285-8b64-da272361f851-service-ca\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.408159 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.408067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2f00827-2629-4285-8b64-da272361f851-oauth-serving-cert\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.408159 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.408097 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2f00827-2629-4285-8b64-da272361f851-console-config\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.408485 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.408465 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2f00827-2629-4285-8b64-da272361f851-trusted-ca-bundle\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.409603 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.409581 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2f00827-2629-4285-8b64-da272361f851-console-serving-cert\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.409741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.409725 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2f00827-2629-4285-8b64-da272361f851-console-oauth-config\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.416078 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.416054 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq8qz\" (UniqueName: \"kubernetes.io/projected/b2f00827-2629-4285-8b64-da272361f851-kube-api-access-cq8qz\") pod \"console-5b756d7cd5-fdlsn\" (UID: \"b2f00827-2629-4285-8b64-da272361f851\") " pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.533709 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.533593 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:43.675800 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.675747 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b756d7cd5-fdlsn"] Jul 23 00:34:43.678457 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:34:43.678431 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f00827_2629_4285_8b64_da272361f851.slice/crio-c66b562a8a0fbd335a20e7c68ce9aa3d03f70d80926c08ccb1ef2a145208d281 WatchSource:0}: Error finding container c66b562a8a0fbd335a20e7c68ce9aa3d03f70d80926c08ccb1ef2a145208d281: Status 404 returned error can't find the container with id c66b562a8a0fbd335a20e7c68ce9aa3d03f70d80926c08ccb1ef2a145208d281 Jul 23 00:34:43.941391 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.941355 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b756d7cd5-fdlsn" event={"ID":"b2f00827-2629-4285-8b64-da272361f851","Type":"ContainerStarted","Data":"49950e3b5d1eb601454bbe0ab63a9af13b2a61937e56d92263a75950acfbed3e"} Jul 23 00:34:43.941657 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.941414 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b756d7cd5-fdlsn" event={"ID":"b2f00827-2629-4285-8b64-da272361f851","Type":"ContainerStarted","Data":"c66b562a8a0fbd335a20e7c68ce9aa3d03f70d80926c08ccb1ef2a145208d281"} Jul 23 00:34:43.960228 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:43.960168 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b756d7cd5-fdlsn" podStartSLOduration=0.960148746 podStartE2EDuration="960.148746ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:34:43.958894948 +0000 UTC m=+676.454563661" watchObservedRunningTime="2026-07-23 00:34:43.960148746 +0000 UTC m=+676.455817533" Jul 23 00:34:45.441418 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:45.441380 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:48.840137 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:48.840096 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:49.646028 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:49.645990 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:52.913113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:52.913075 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:53.023134 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:53.023093 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:53.341993 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:53.341903 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:53.534086 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:53.534044 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:53.534263 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:53.534105 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:53.539355 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:53.539327 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:53.773913 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:53.773878 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:53.998880 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:53.998851 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b756d7cd5-fdlsn" Jul 23 00:34:54.052121 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:54.052040 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-786675b496-lgplc"] Jul 23 00:34:54.226162 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:54.226123 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:54.388918 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:54.388881 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:54.866575 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:54.866434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:56.101994 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:56.101948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:56.189787 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:56.189741 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:34:57.928213 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:34:57.928172 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:01.053459 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:01.053410 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:01.183309 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:01.183270 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:06.439188 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:06.439152 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:06.542316 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:06.542278 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:06.966347 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:06.966309 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:07.080688 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:07.080651 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:10.186280 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:10.186238 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:10.444523 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:10.444413 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:10.698693 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:10.698604 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:10.799998 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:10.799967 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:14.196432 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:14.196392 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:14.296425 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:14.296386 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:18.711368 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:18.711331 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:18.811579 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:18.811522 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:19.073822 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.073729 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-786675b496-lgplc" podUID="6aefe260-fb1f-4495-8d24-1f11d952fe58" containerName="console" containerID="cri-o://1f074ee1528398706b1f53ead35fb04fe90bc6e7202200133b1f6e80cc352340" gracePeriod=15 Jul 23 00:35:19.317040 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.317013 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786675b496-lgplc" Jul 23 00:35:19.441830 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.441780 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-config\") pod \"6aefe260-fb1f-4495-8d24-1f11d952fe58\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " Jul 23 00:35:19.441830 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.441853 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-oauth-serving-cert\") pod \"6aefe260-fb1f-4495-8d24-1f11d952fe58\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " Jul 23 00:35:19.442124 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.441887 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7w9m\" (UniqueName: \"kubernetes.io/projected/6aefe260-fb1f-4495-8d24-1f11d952fe58-kube-api-access-d7w9m\") pod \"6aefe260-fb1f-4495-8d24-1f11d952fe58\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " Jul 23 00:35:19.442124 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.441910 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-oauth-config\") pod \"6aefe260-fb1f-4495-8d24-1f11d952fe58\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " Jul 23 00:35:19.442124 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.442012 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-serving-cert\") pod \"6aefe260-fb1f-4495-8d24-1f11d952fe58\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " Jul 23 00:35:19.442124 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.442051 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-trusted-ca-bundle\") pod \"6aefe260-fb1f-4495-8d24-1f11d952fe58\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " Jul 23 00:35:19.442124 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.442084 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-service-ca\") pod \"6aefe260-fb1f-4495-8d24-1f11d952fe58\" (UID: \"6aefe260-fb1f-4495-8d24-1f11d952fe58\") " Jul 23 00:35:19.442416 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.442296 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-config" (OuterVolumeSpecName: "console-config") pod "6aefe260-fb1f-4495-8d24-1f11d952fe58" (UID: "6aefe260-fb1f-4495-8d24-1f11d952fe58"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:35:19.442484 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.442301 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6aefe260-fb1f-4495-8d24-1f11d952fe58" (UID: "6aefe260-fb1f-4495-8d24-1f11d952fe58"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:35:19.442609 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.442580 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6aefe260-fb1f-4495-8d24-1f11d952fe58" (UID: "6aefe260-fb1f-4495-8d24-1f11d952fe58"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:35:19.442783 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.442761 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-service-ca" (OuterVolumeSpecName: "service-ca") pod "6aefe260-fb1f-4495-8d24-1f11d952fe58" (UID: "6aefe260-fb1f-4495-8d24-1f11d952fe58"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:35:19.444262 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.444238 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6aefe260-fb1f-4495-8d24-1f11d952fe58" (UID: "6aefe260-fb1f-4495-8d24-1f11d952fe58"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:35:19.444647 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.444622 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6aefe260-fb1f-4495-8d24-1f11d952fe58" (UID: "6aefe260-fb1f-4495-8d24-1f11d952fe58"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:35:19.444716 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.444658 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aefe260-fb1f-4495-8d24-1f11d952fe58-kube-api-access-d7w9m" (OuterVolumeSpecName: "kube-api-access-d7w9m") pod "6aefe260-fb1f-4495-8d24-1f11d952fe58" (UID: "6aefe260-fb1f-4495-8d24-1f11d952fe58"). InnerVolumeSpecName "kube-api-access-d7w9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:35:19.543553 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.543415 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-serving-cert\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:35:19.543553 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.543467 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-trusted-ca-bundle\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:35:19.543553 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.543477 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:35:19.543553 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.543486 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-config\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:35:19.543553 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.543494 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6aefe260-fb1f-4495-8d24-1f11d952fe58-oauth-serving-cert\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:35:19.543553 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.543503 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7w9m\" (UniqueName: \"kubernetes.io/projected/6aefe260-fb1f-4495-8d24-1f11d952fe58-kube-api-access-d7w9m\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:35:19.543553 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:19.543511 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6aefe260-fb1f-4495-8d24-1f11d952fe58-console-oauth-config\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:35:20.140650 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:20.140612 2564 generic.go:356] "Generic (PLEG): container finished" podID="6aefe260-fb1f-4495-8d24-1f11d952fe58" containerID="1f074ee1528398706b1f53ead35fb04fe90bc6e7202200133b1f6e80cc352340" exitCode=0 Jul 23 00:35:20.141135 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:20.140657 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-786675b496-lgplc" Jul 23 00:35:20.141135 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:20.140673 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786675b496-lgplc" event={"ID":"6aefe260-fb1f-4495-8d24-1f11d952fe58","Type":"ContainerDied","Data":"1f074ee1528398706b1f53ead35fb04fe90bc6e7202200133b1f6e80cc352340"} Jul 23 00:35:20.141135 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:20.140725 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-786675b496-lgplc" event={"ID":"6aefe260-fb1f-4495-8d24-1f11d952fe58","Type":"ContainerDied","Data":"cc816fdf3545cbdd1c88c70b5f904b7e01bd6e32aa2dbe783025a249c47737a9"} Jul 23 00:35:20.141135 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:20.140746 2564 scope.go:117] "RemoveContainer" containerID="1f074ee1528398706b1f53ead35fb04fe90bc6e7202200133b1f6e80cc352340" Jul 23 00:35:20.151120 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:20.151058 2564 scope.go:117] "RemoveContainer" containerID="1f074ee1528398706b1f53ead35fb04fe90bc6e7202200133b1f6e80cc352340" Jul 23 00:35:20.152042 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:35:20.152007 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f074ee1528398706b1f53ead35fb04fe90bc6e7202200133b1f6e80cc352340\": container with ID starting with 1f074ee1528398706b1f53ead35fb04fe90bc6e7202200133b1f6e80cc352340 not found: ID does not exist" containerID="1f074ee1528398706b1f53ead35fb04fe90bc6e7202200133b1f6e80cc352340" Jul 23 00:35:20.152179 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:20.152051 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f074ee1528398706b1f53ead35fb04fe90bc6e7202200133b1f6e80cc352340"} err="failed to get container status \"1f074ee1528398706b1f53ead35fb04fe90bc6e7202200133b1f6e80cc352340\": rpc error: code = NotFound desc = could not find container \"1f074ee1528398706b1f53ead35fb04fe90bc6e7202200133b1f6e80cc352340\": container with ID starting with 1f074ee1528398706b1f53ead35fb04fe90bc6e7202200133b1f6e80cc352340 not found: ID does not exist" Jul 23 00:35:20.166872 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:20.166836 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-786675b496-lgplc"] Jul 23 00:35:20.171691 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:20.171658 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-786675b496-lgplc"] Jul 23 00:35:20.941670 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:20.941635 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:21.044302 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:21.044268 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:22.073693 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:22.073657 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aefe260-fb1f-4495-8d24-1f11d952fe58" path="/var/lib/kubelet/pods/6aefe260-fb1f-4495-8d24-1f11d952fe58/volumes" Jul 23 00:35:24.627556 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:24.627486 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:24.712062 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:24.712021 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:24.860989 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:24.860950 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:24.923679 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:24.923632 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:28.426289 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:28.426250 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:28.526679 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:28.526639 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:32.992885 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:32.992852 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:33.094544 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:33.094490 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:35:45.677114 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:35:45.677071 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:05.853788 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.853751 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx"] Jul 23 00:36:05.854290 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.854138 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aefe260-fb1f-4495-8d24-1f11d952fe58" containerName="console" Jul 23 00:36:05.854290 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.854149 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aefe260-fb1f-4495-8d24-1f11d952fe58" containerName="console" Jul 23 00:36:05.854290 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.854238 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aefe260-fb1f-4495-8d24-1f11d952fe58" containerName="console" Jul 23 00:36:05.857332 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.857310 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:05.860982 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.860936 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sbkcg\"" Jul 23 00:36:05.889170 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.889130 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx"] Jul 23 00:36:05.927172 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.927137 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:05.952288 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.952247 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b3b006-3ea6-445d-bf61-b1fed46711f0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:05.952288 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.952286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sklkv\" (UniqueName: \"kubernetes.io/projected/d5b3b006-3ea6-445d-bf61-b1fed46711f0-kube-api-access-sklkv\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:05.952518 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.952334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b3b006-3ea6-445d-bf61-b1fed46711f0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:05.952518 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.952359 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b3b006-3ea6-445d-bf61-b1fed46711f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:05.952518 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.952396 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b3b006-3ea6-445d-bf61-b1fed46711f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:05.952518 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.952478 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b3b006-3ea6-445d-bf61-b1fed46711f0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:05.952518 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.952494 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b3b006-3ea6-445d-bf61-b1fed46711f0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:05.952731 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.952559 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b3b006-3ea6-445d-bf61-b1fed46711f0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:05.952731 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:05.952597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b3b006-3ea6-445d-bf61-b1fed46711f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.053543 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.053493 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b3b006-3ea6-445d-bf61-b1fed46711f0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.053736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.053587 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b3b006-3ea6-445d-bf61-b1fed46711f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.053736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.053612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b3b006-3ea6-445d-bf61-b1fed46711f0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.053736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.053628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sklkv\" (UniqueName: \"kubernetes.io/projected/d5b3b006-3ea6-445d-bf61-b1fed46711f0-kube-api-access-sklkv\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.053736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.053656 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b3b006-3ea6-445d-bf61-b1fed46711f0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.053736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.053690 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b3b006-3ea6-445d-bf61-b1fed46711f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.054009 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.053754 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b3b006-3ea6-445d-bf61-b1fed46711f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.054009 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.053965 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b3b006-3ea6-445d-bf61-b1fed46711f0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.054009 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.054004 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5b3b006-3ea6-445d-bf61-b1fed46711f0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.054152 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.054100 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b3b006-3ea6-445d-bf61-b1fed46711f0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.054152 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.054123 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b3b006-3ea6-445d-bf61-b1fed46711f0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.054477 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.054445 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5b3b006-3ea6-445d-bf61-b1fed46711f0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.054477 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.054469 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5b3b006-3ea6-445d-bf61-b1fed46711f0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.054937 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.054907 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5b3b006-3ea6-445d-bf61-b1fed46711f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.056182 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.056158 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5b3b006-3ea6-445d-bf61-b1fed46711f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.056403 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.056386 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5b3b006-3ea6-445d-bf61-b1fed46711f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.062921 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.062898 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5b3b006-3ea6-445d-bf61-b1fed46711f0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.063095 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.063074 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sklkv\" (UniqueName: \"kubernetes.io/projected/d5b3b006-3ea6-445d-bf61-b1fed46711f0-kube-api-access-sklkv\") pod \"maas-default-gateway-openshift-default-56654dcf7-9srtx\" (UID: \"d5b3b006-3ea6-445d-bf61-b1fed46711f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.146218 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.146173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:06.168657 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.168621 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:06.315636 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.315601 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx"] Jul 23 00:36:06.316959 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:36:06.316935 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b3b006_3ea6_445d_bf61_b1fed46711f0.slice/crio-4e0940cac652c2107c887fb1007725174e445f5f0c6dc666a20951deaa63fb31 WatchSource:0}: Error finding container 4e0940cac652c2107c887fb1007725174e445f5f0c6dc666a20951deaa63fb31: Status 404 returned error can't find the container with id 4e0940cac652c2107c887fb1007725174e445f5f0c6dc666a20951deaa63fb31 Jul 23 00:36:06.319129 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.319084 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 00:36:06.319231 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.319205 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 00:36:06.319294 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.319269 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 00:36:06.398518 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.398415 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" event={"ID":"d5b3b006-3ea6-445d-bf61-b1fed46711f0","Type":"ContainerStarted","Data":"1386930b8d5f6d14c3468b51b6c2d96eb24f4bcae49ddd8b71f2e5eb7d4e4eca"} Jul 23 00:36:06.398518 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.398471 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" event={"ID":"d5b3b006-3ea6-445d-bf61-b1fed46711f0","Type":"ContainerStarted","Data":"4e0940cac652c2107c887fb1007725174e445f5f0c6dc666a20951deaa63fb31"} Jul 23 00:36:06.419282 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:06.419226 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" podStartSLOduration=1.419208721 podStartE2EDuration="1.419208721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:36:06.417865975 +0000 UTC m=+758.913534689" watchObservedRunningTime="2026-07-23 00:36:06.419208721 +0000 UTC m=+758.914877434" Jul 23 00:36:07.168894 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:07.168857 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:07.174002 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:07.173972 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:07.404429 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:07.404400 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:07.405482 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:07.405463 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:36:10.191265 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:10.191231 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:10.297086 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:10.297047 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:10.400299 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:10.400260 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:10.992822 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:10.992781 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-dn56l"] Jul 23 00:36:10.996326 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:10.996302 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dn56l" Jul 23 00:36:10.998992 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:10.998966 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 00:36:10.999138 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:10.998966 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 00:36:11.000164 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:11.000144 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sx4qg\"" Jul 23 00:36:11.003646 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:11.003621 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-dn56l"] Jul 23 00:36:11.029626 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:11.029579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:11.102995 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:11.102960 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n8w8\" (UniqueName: \"kubernetes.io/projected/d8d238d0-2a27-4a89-a1fa-721a4360af2a-kube-api-access-6n8w8\") pod \"authorino-58cd955675-dn56l\" (UID: \"d8d238d0-2a27-4a89-a1fa-721a4360af2a\") " pod="kuadrant-system/authorino-58cd955675-dn56l" Jul 23 00:36:11.130136 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:11.130106 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:11.204166 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:11.204136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n8w8\" (UniqueName: \"kubernetes.io/projected/d8d238d0-2a27-4a89-a1fa-721a4360af2a-kube-api-access-6n8w8\") pod \"authorino-58cd955675-dn56l\" (UID: \"d8d238d0-2a27-4a89-a1fa-721a4360af2a\") " pod="kuadrant-system/authorino-58cd955675-dn56l" Jul 23 00:36:11.213795 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:11.213760 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n8w8\" (UniqueName: \"kubernetes.io/projected/d8d238d0-2a27-4a89-a1fa-721a4360af2a-kube-api-access-6n8w8\") pod \"authorino-58cd955675-dn56l\" (UID: \"d8d238d0-2a27-4a89-a1fa-721a4360af2a\") " pod="kuadrant-system/authorino-58cd955675-dn56l" Jul 23 00:36:11.308566 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:11.308453 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dn56l" Jul 23 00:36:11.453491 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:11.453467 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-dn56l"] Jul 23 00:36:11.455368 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:36:11.455340 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8d238d0_2a27_4a89_a1fa_721a4360af2a.slice/crio-d0a044c8a4e1a988b93bed66818c54f2da34ca86b37bc277ac5597b628e8006c WatchSource:0}: Error finding container d0a044c8a4e1a988b93bed66818c54f2da34ca86b37bc277ac5597b628e8006c: Status 404 returned error can't find the container with id d0a044c8a4e1a988b93bed66818c54f2da34ca86b37bc277ac5597b628e8006c Jul 23 00:36:12.436741 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:12.436704 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dn56l" event={"ID":"d8d238d0-2a27-4a89-a1fa-721a4360af2a","Type":"ContainerStarted","Data":"d0a044c8a4e1a988b93bed66818c54f2da34ca86b37bc277ac5597b628e8006c"} Jul 23 00:36:15.455514 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:15.455474 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dn56l" event={"ID":"d8d238d0-2a27-4a89-a1fa-721a4360af2a","Type":"ContainerStarted","Data":"7342cae9c809437af5a8d76d5733e4f6a5e3b9cfcb79dcd2afff852ad33a90a0"} Jul 23 00:36:15.471823 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:15.471637 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-dn56l" podStartSLOduration=2.035631445 podStartE2EDuration="5.471615079s" totalImagesPullingTime="3.435983634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:36:15.469328888 +0000 UTC m=+767.964997603" watchObservedRunningTime="2026-07-23 00:36:15.471615079 +0000 UTC m=+767.967283794" Jul 23 00:36:15.720041 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:15.719946 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-dn56l"] Jul 23 00:36:17.465570 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:17.465514 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-dn56l" podUID="d8d238d0-2a27-4a89-a1fa-721a4360af2a" containerName="authorino" containerID="cri-o://7342cae9c809437af5a8d76d5733e4f6a5e3b9cfcb79dcd2afff852ad33a90a0" gracePeriod=30 Jul 23 00:36:17.731565 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:17.731472 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dn56l" Jul 23 00:36:17.867845 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:17.867807 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n8w8\" (UniqueName: \"kubernetes.io/projected/d8d238d0-2a27-4a89-a1fa-721a4360af2a-kube-api-access-6n8w8\") pod \"d8d238d0-2a27-4a89-a1fa-721a4360af2a\" (UID: \"d8d238d0-2a27-4a89-a1fa-721a4360af2a\") " Jul 23 00:36:17.869926 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:17.869899 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d238d0-2a27-4a89-a1fa-721a4360af2a-kube-api-access-6n8w8" (OuterVolumeSpecName: "kube-api-access-6n8w8") pod "d8d238d0-2a27-4a89-a1fa-721a4360af2a" (UID: "d8d238d0-2a27-4a89-a1fa-721a4360af2a"). InnerVolumeSpecName "kube-api-access-6n8w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:36:17.969432 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:17.969387 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n8w8\" (UniqueName: \"kubernetes.io/projected/d8d238d0-2a27-4a89-a1fa-721a4360af2a-kube-api-access-6n8w8\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:36:18.471909 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:18.471876 2564 generic.go:356] "Generic (PLEG): container finished" podID="d8d238d0-2a27-4a89-a1fa-721a4360af2a" containerID="7342cae9c809437af5a8d76d5733e4f6a5e3b9cfcb79dcd2afff852ad33a90a0" exitCode=0 Jul 23 00:36:18.472413 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:18.471930 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-dn56l" Jul 23 00:36:18.472413 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:18.471975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dn56l" event={"ID":"d8d238d0-2a27-4a89-a1fa-721a4360af2a","Type":"ContainerDied","Data":"7342cae9c809437af5a8d76d5733e4f6a5e3b9cfcb79dcd2afff852ad33a90a0"} Jul 23 00:36:18.472413 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:18.472026 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-dn56l" event={"ID":"d8d238d0-2a27-4a89-a1fa-721a4360af2a","Type":"ContainerDied","Data":"d0a044c8a4e1a988b93bed66818c54f2da34ca86b37bc277ac5597b628e8006c"} Jul 23 00:36:18.472413 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:18.472045 2564 scope.go:117] "RemoveContainer" containerID="7342cae9c809437af5a8d76d5733e4f6a5e3b9cfcb79dcd2afff852ad33a90a0" Jul 23 00:36:18.481082 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:18.481059 2564 scope.go:117] "RemoveContainer" containerID="7342cae9c809437af5a8d76d5733e4f6a5e3b9cfcb79dcd2afff852ad33a90a0" Jul 23 00:36:18.481422 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:36:18.481400 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7342cae9c809437af5a8d76d5733e4f6a5e3b9cfcb79dcd2afff852ad33a90a0\": container with ID starting with 7342cae9c809437af5a8d76d5733e4f6a5e3b9cfcb79dcd2afff852ad33a90a0 not found: ID does not exist" containerID="7342cae9c809437af5a8d76d5733e4f6a5e3b9cfcb79dcd2afff852ad33a90a0" Jul 23 00:36:18.481480 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:18.481432 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7342cae9c809437af5a8d76d5733e4f6a5e3b9cfcb79dcd2afff852ad33a90a0"} err="failed to get container status \"7342cae9c809437af5a8d76d5733e4f6a5e3b9cfcb79dcd2afff852ad33a90a0\": rpc error: code = NotFound desc = could not find container \"7342cae9c809437af5a8d76d5733e4f6a5e3b9cfcb79dcd2afff852ad33a90a0\": container with ID starting with 7342cae9c809437af5a8d76d5733e4f6a5e3b9cfcb79dcd2afff852ad33a90a0 not found: ID does not exist" Jul 23 00:36:18.489060 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:18.489028 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-dn56l"] Jul 23 00:36:18.492487 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:18.492461 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-dn56l"] Jul 23 00:36:20.073688 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:20.073651 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8d238d0-2a27-4a89-a1fa-721a4360af2a" path="/var/lib/kubelet/pods/d8d238d0-2a27-4a89-a1fa-721a4360af2a/volumes" Jul 23 00:36:28.654426 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:28.654385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:28.746190 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:28.746145 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:29.112970 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:29.112886 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:34.410285 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:34.410245 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:34.511138 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:34.511093 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:39.842686 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:39.842641 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:41.031250 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.031215 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:41.132021 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.131985 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:41.262110 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.262071 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6gjf8"] Jul 23 00:36:41.262506 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.262493 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8d238d0-2a27-4a89-a1fa-721a4360af2a" containerName="authorino" Jul 23 00:36:41.262600 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.262508 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8d238d0-2a27-4a89-a1fa-721a4360af2a" containerName="authorino" Jul 23 00:36:41.262638 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.262607 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8d238d0-2a27-4a89-a1fa-721a4360af2a" containerName="authorino" Jul 23 00:36:41.266886 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.266864 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6gjf8" Jul 23 00:36:41.270173 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.270149 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 00:36:41.270330 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.270215 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 00:36:41.271278 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.271256 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sx4qg\"" Jul 23 00:36:41.273412 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.273391 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6gjf8"] Jul 23 00:36:41.292844 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.292750 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgx7b\" (UniqueName: \"kubernetes.io/projected/9c1f99e2-3a72-47de-8ec4-8a1899229c06-kube-api-access-cgx7b\") pod \"authorino-5d4b67c446-6gjf8\" (UID: \"9c1f99e2-3a72-47de-8ec4-8a1899229c06\") " pod="kuadrant-system/authorino-5d4b67c446-6gjf8" Jul 23 00:36:41.393948 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.393891 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgx7b\" (UniqueName: \"kubernetes.io/projected/9c1f99e2-3a72-47de-8ec4-8a1899229c06-kube-api-access-cgx7b\") pod \"authorino-5d4b67c446-6gjf8\" (UID: \"9c1f99e2-3a72-47de-8ec4-8a1899229c06\") " pod="kuadrant-system/authorino-5d4b67c446-6gjf8" Jul 23 00:36:41.402974 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.402940 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgx7b\" (UniqueName: \"kubernetes.io/projected/9c1f99e2-3a72-47de-8ec4-8a1899229c06-kube-api-access-cgx7b\") pod \"authorino-5d4b67c446-6gjf8\" (UID: \"9c1f99e2-3a72-47de-8ec4-8a1899229c06\") " pod="kuadrant-system/authorino-5d4b67c446-6gjf8" Jul 23 00:36:41.532835 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.532792 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6gjf8"] Jul 23 00:36:41.533100 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.533088 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6gjf8" Jul 23 00:36:41.552369 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.552336 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-797974f89-whc5d"] Jul 23 00:36:41.555008 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.554982 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-797974f89-whc5d" Jul 23 00:36:41.557490 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.557461 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 23 00:36:41.565213 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.565188 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-797974f89-whc5d"] Jul 23 00:36:41.596253 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.596211 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2wsb\" (UniqueName: \"kubernetes.io/projected/6b1c6037-ff7a-4484-86f4-c3a90813bd18-kube-api-access-r2wsb\") pod \"authorino-797974f89-whc5d\" (UID: \"6b1c6037-ff7a-4484-86f4-c3a90813bd18\") " pod="kuadrant-system/authorino-797974f89-whc5d" Jul 23 00:36:41.596403 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.596384 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6b1c6037-ff7a-4484-86f4-c3a90813bd18-tls-cert\") pod \"authorino-797974f89-whc5d\" (UID: \"6b1c6037-ff7a-4484-86f4-c3a90813bd18\") " pod="kuadrant-system/authorino-797974f89-whc5d" Jul 23 00:36:41.676939 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.676913 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6gjf8"] Jul 23 00:36:41.679305 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:36:41.679275 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c1f99e2_3a72_47de_8ec4_8a1899229c06.slice/crio-6b39957f8002feda2e2c691f43e2ec29c093b1d90c6eb83860c98414f512bb3b WatchSource:0}: Error finding container 6b39957f8002feda2e2c691f43e2ec29c093b1d90c6eb83860c98414f512bb3b: Status 404 returned error can't find the container with id 6b39957f8002feda2e2c691f43e2ec29c093b1d90c6eb83860c98414f512bb3b Jul 23 00:36:41.697708 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.697660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6b1c6037-ff7a-4484-86f4-c3a90813bd18-tls-cert\") pod \"authorino-797974f89-whc5d\" (UID: \"6b1c6037-ff7a-4484-86f4-c3a90813bd18\") " pod="kuadrant-system/authorino-797974f89-whc5d" Jul 23 00:36:41.697885 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.697812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2wsb\" (UniqueName: \"kubernetes.io/projected/6b1c6037-ff7a-4484-86f4-c3a90813bd18-kube-api-access-r2wsb\") pod \"authorino-797974f89-whc5d\" (UID: \"6b1c6037-ff7a-4484-86f4-c3a90813bd18\") " pod="kuadrant-system/authorino-797974f89-whc5d" Jul 23 00:36:41.700198 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.700166 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6b1c6037-ff7a-4484-86f4-c3a90813bd18-tls-cert\") pod \"authorino-797974f89-whc5d\" (UID: \"6b1c6037-ff7a-4484-86f4-c3a90813bd18\") " pod="kuadrant-system/authorino-797974f89-whc5d" Jul 23 00:36:41.706768 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.706740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2wsb\" (UniqueName: \"kubernetes.io/projected/6b1c6037-ff7a-4484-86f4-c3a90813bd18-kube-api-access-r2wsb\") pod \"authorino-797974f89-whc5d\" (UID: \"6b1c6037-ff7a-4484-86f4-c3a90813bd18\") " pod="kuadrant-system/authorino-797974f89-whc5d" Jul 23 00:36:41.867388 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:41.867292 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-797974f89-whc5d" Jul 23 00:36:42.006201 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:42.006171 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-797974f89-whc5d"] Jul 23 00:36:42.008115 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:36:42.008089 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b1c6037_ff7a_4484_86f4_c3a90813bd18.slice/crio-8426c608fbec18c5b853ea31c3aa79fccc22cc9df6dacd1360d2d9c40cc0c7a6 WatchSource:0}: Error finding container 8426c608fbec18c5b853ea31c3aa79fccc22cc9df6dacd1360d2d9c40cc0c7a6: Status 404 returned error can't find the container with id 8426c608fbec18c5b853ea31c3aa79fccc22cc9df6dacd1360d2d9c40cc0c7a6 Jul 23 00:36:42.611312 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:42.611204 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-797974f89-whc5d" event={"ID":"6b1c6037-ff7a-4484-86f4-c3a90813bd18","Type":"ContainerStarted","Data":"6e6d1027baef46f0f5279b51e30f5db23329c9fd2f4fcfa76f459996e3c1bb79"} Jul 23 00:36:42.611312 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:42.611255 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-797974f89-whc5d" event={"ID":"6b1c6037-ff7a-4484-86f4-c3a90813bd18","Type":"ContainerStarted","Data":"8426c608fbec18c5b853ea31c3aa79fccc22cc9df6dacd1360d2d9c40cc0c7a6"} Jul 23 00:36:42.612306 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:42.612284 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6gjf8" event={"ID":"9c1f99e2-3a72-47de-8ec4-8a1899229c06","Type":"ContainerStarted","Data":"6b39957f8002feda2e2c691f43e2ec29c093b1d90c6eb83860c98414f512bb3b"} Jul 23 00:36:42.629789 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:42.629725 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-797974f89-whc5d" podStartSLOduration=1.3304588 podStartE2EDuration="1.629707829s" totalImagesPullingTime="299.249029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:36:42.626751431 +0000 UTC m=+795.122420165" watchObservedRunningTime="2026-07-23 00:36:42.629707829 +0000 UTC m=+795.125376519" Jul 23 00:36:44.630743 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:44.630590 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6gjf8" event={"ID":"9c1f99e2-3a72-47de-8ec4-8a1899229c06","Type":"ContainerStarted","Data":"8baaf63fe40cd9ae5e86511f796127b9005fda3ce28eb0a3f5759e0da4446739"} Jul 23 00:36:44.631403 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:44.630818 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-6gjf8" podUID="9c1f99e2-3a72-47de-8ec4-8a1899229c06" containerName="authorino" containerID="cri-o://8baaf63fe40cd9ae5e86511f796127b9005fda3ce28eb0a3f5759e0da4446739" gracePeriod=30 Jul 23 00:36:44.649294 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:44.649229 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-6gjf8" podStartSLOduration=1.1184938739999999 podStartE2EDuration="3.6492062s" totalImagesPullingTime="2.530712326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:36:44.645730055 +0000 UTC m=+797.141398793" watchObservedRunningTime="2026-07-23 00:36:44.6492062 +0000 UTC m=+797.144874918" Jul 23 00:36:44.891565 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:44.891510 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6gjf8" Jul 23 00:36:44.933648 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:44.933613 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgx7b\" (UniqueName: \"kubernetes.io/projected/9c1f99e2-3a72-47de-8ec4-8a1899229c06-kube-api-access-cgx7b\") pod \"9c1f99e2-3a72-47de-8ec4-8a1899229c06\" (UID: \"9c1f99e2-3a72-47de-8ec4-8a1899229c06\") " Jul 23 00:36:44.935649 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:44.935615 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1f99e2-3a72-47de-8ec4-8a1899229c06-kube-api-access-cgx7b" (OuterVolumeSpecName: "kube-api-access-cgx7b") pod "9c1f99e2-3a72-47de-8ec4-8a1899229c06" (UID: "9c1f99e2-3a72-47de-8ec4-8a1899229c06"). InnerVolumeSpecName "kube-api-access-cgx7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:36:45.035228 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:45.035185 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgx7b\" (UniqueName: \"kubernetes.io/projected/9c1f99e2-3a72-47de-8ec4-8a1899229c06-kube-api-access-cgx7b\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:36:45.636845 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:45.636815 2564 generic.go:356] "Generic (PLEG): container finished" podID="9c1f99e2-3a72-47de-8ec4-8a1899229c06" containerID="8baaf63fe40cd9ae5e86511f796127b9005fda3ce28eb0a3f5759e0da4446739" exitCode=0 Jul 23 00:36:45.637269 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:45.636860 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6gjf8" Jul 23 00:36:45.637269 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:45.636901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6gjf8" event={"ID":"9c1f99e2-3a72-47de-8ec4-8a1899229c06","Type":"ContainerDied","Data":"8baaf63fe40cd9ae5e86511f796127b9005fda3ce28eb0a3f5759e0da4446739"} Jul 23 00:36:45.637269 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:45.636940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6gjf8" event={"ID":"9c1f99e2-3a72-47de-8ec4-8a1899229c06","Type":"ContainerDied","Data":"6b39957f8002feda2e2c691f43e2ec29c093b1d90c6eb83860c98414f512bb3b"} Jul 23 00:36:45.637269 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:45.636956 2564 scope.go:117] "RemoveContainer" containerID="8baaf63fe40cd9ae5e86511f796127b9005fda3ce28eb0a3f5759e0da4446739" Jul 23 00:36:45.646673 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:45.646625 2564 scope.go:117] "RemoveContainer" containerID="8baaf63fe40cd9ae5e86511f796127b9005fda3ce28eb0a3f5759e0da4446739" Jul 23 00:36:45.646903 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:36:45.646883 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8baaf63fe40cd9ae5e86511f796127b9005fda3ce28eb0a3f5759e0da4446739\": container with ID starting with 8baaf63fe40cd9ae5e86511f796127b9005fda3ce28eb0a3f5759e0da4446739 not found: ID does not exist" containerID="8baaf63fe40cd9ae5e86511f796127b9005fda3ce28eb0a3f5759e0da4446739" Jul 23 00:36:45.646946 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:45.646912 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8baaf63fe40cd9ae5e86511f796127b9005fda3ce28eb0a3f5759e0da4446739"} err="failed to get container status \"8baaf63fe40cd9ae5e86511f796127b9005fda3ce28eb0a3f5759e0da4446739\": rpc error: code = NotFound desc = could not find container \"8baaf63fe40cd9ae5e86511f796127b9005fda3ce28eb0a3f5759e0da4446739\": container with ID starting with 8baaf63fe40cd9ae5e86511f796127b9005fda3ce28eb0a3f5759e0da4446739 not found: ID does not exist" Jul 23 00:36:45.659408 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:45.659387 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6gjf8"] Jul 23 00:36:45.663081 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:45.663059 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6gjf8"] Jul 23 00:36:46.073150 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:46.073065 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c1f99e2-3a72-47de-8ec4-8a1899229c06" path="/var/lib/kubelet/pods/9c1f99e2-3a72-47de-8ec4-8a1899229c06/volumes" Jul 23 00:36:47.387292 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:47.387255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:47.456175 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:47.456136 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:47.546337 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:47.546305 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:47.897117 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:47.897078 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:51.142242 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:51.142151 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:36:51.254137 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:36:51.254094 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:37:06.650249 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:06.650215 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:37:06.708428 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:06.708389 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:37:23.170879 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:23.170842 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:37:23.271005 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:23.270964 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:37:23.829215 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:23.829176 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:37:24.083186 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:24.083099 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:37:24.408838 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:24.408798 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:37:24.508161 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:24.508121 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:37:32.904728 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:32.904696 2564 generic.go:356] "Generic (PLEG): container finished" podID="d5b3b006-3ea6-445d-bf61-b1fed46711f0" containerID="1386930b8d5f6d14c3468b51b6c2d96eb24f4bcae49ddd8b71f2e5eb7d4e4eca" exitCode=0 Jul 23 00:37:32.905290 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:32.904745 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" event={"ID":"d5b3b006-3ea6-445d-bf61-b1fed46711f0","Type":"ContainerDied","Data":"1386930b8d5f6d14c3468b51b6c2d96eb24f4bcae49ddd8b71f2e5eb7d4e4eca"} Jul 23 00:37:32.905290 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:32.905251 2564 scope.go:117] "RemoveContainer" containerID="1386930b8d5f6d14c3468b51b6c2d96eb24f4bcae49ddd8b71f2e5eb7d4e4eca" Jul 23 00:37:32.906351 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:32.906317 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 00:37:32.906415 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:32.906392 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 00:37:32.906492 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:32.906475 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 00:37:33.912940 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:33.912898 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" event={"ID":"d5b3b006-3ea6-445d-bf61-b1fed46711f0","Type":"ContainerStarted","Data":"e59eb11dda02c96e50586a537263b13213af366b36c525219e9b6db9ebb84fcb"} Jul 23 00:37:34.169771 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:34.169677 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:37:34.170933 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:34.170898 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-9srtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 00:37:34.171083 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:34.170952 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" podUID="d5b3b006-3ea6-445d-bf61-b1fed46711f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 00:37:34.961122 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:34.961081 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:37:35.055566 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:35.055503 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:37:35.169923 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:35.169886 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-9srtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 00:37:35.170106 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:35.169950 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" podUID="d5b3b006-3ea6-445d-bf61-b1fed46711f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 00:37:35.608150 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:35.608111 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:37:36.169556 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:36.169478 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-9srtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 00:37:36.170079 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:36.169571 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" podUID="d5b3b006-3ea6-445d-bf61-b1fed46711f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 00:37:37.169802 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:37.169766 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-9srtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 00:37:37.170207 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:37.169834 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" podUID="d5b3b006-3ea6-445d-bf61-b1fed46711f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 00:37:37.405222 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:37.405176 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:37:38.173474 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:38.173439 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:37:38.174865 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:38.174836 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-9srtx" Jul 23 00:37:55.700587 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.700550 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9"] Jul 23 00:37:55.701011 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.700980 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c1f99e2-3a72-47de-8ec4-8a1899229c06" containerName="authorino" Jul 23 00:37:55.701011 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.700992 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c1f99e2-3a72-47de-8ec4-8a1899229c06" containerName="authorino" Jul 23 00:37:55.701089 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.701069 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c1f99e2-3a72-47de-8ec4-8a1899229c06" containerName="authorino" Jul 23 00:37:55.704174 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.704148 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.707169 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.707149 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 00:37:55.707169 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.707149 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 00:37:55.708312 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.708284 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jtgsc\"" Jul 23 00:37:55.708429 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.708399 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 23 00:37:55.716504 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.716478 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9"] Jul 23 00:37:55.730390 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.730360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-985pf\" (UniqueName: \"kubernetes.io/projected/3fce81ce-1e9e-4430-915c-631e557f633f-kube-api-access-985pf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.730567 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.730397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fce81ce-1e9e-4430-915c-631e557f633f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.730567 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.730459 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fce81ce-1e9e-4430-915c-631e557f633f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.730650 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.730572 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fce81ce-1e9e-4430-915c-631e557f633f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.730650 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.730594 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fce81ce-1e9e-4430-915c-631e557f633f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.730650 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.730618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fce81ce-1e9e-4430-915c-631e557f633f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.736047 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.736012 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:37:55.831233 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.831195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fce81ce-1e9e-4430-915c-631e557f633f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.831233 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.831231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fce81ce-1e9e-4430-915c-631e557f633f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.831491 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.831261 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fce81ce-1e9e-4430-915c-631e557f633f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.831491 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.831318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-985pf\" (UniqueName: \"kubernetes.io/projected/3fce81ce-1e9e-4430-915c-631e557f633f-kube-api-access-985pf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.831491 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.831346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fce81ce-1e9e-4430-915c-631e557f633f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.831491 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.831386 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fce81ce-1e9e-4430-915c-631e557f633f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.831742 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.831713 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fce81ce-1e9e-4430-915c-631e557f633f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.831812 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.831788 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fce81ce-1e9e-4430-915c-631e557f633f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.831856 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.831814 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fce81ce-1e9e-4430-915c-631e557f633f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.833675 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.833652 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fce81ce-1e9e-4430-915c-631e557f633f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.833952 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.833933 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fce81ce-1e9e-4430-915c-631e557f633f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:55.839700 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:55.839675 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-985pf\" (UniqueName: \"kubernetes.io/projected/3fce81ce-1e9e-4430-915c-631e557f633f-kube-api-access-985pf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9\" (UID: \"3fce81ce-1e9e-4430-915c-631e557f633f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:56.016284 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:56.016176 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:37:56.164604 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:56.164568 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9"] Jul 23 00:37:56.166081 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:37:56.166050 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fce81ce_1e9e_4430_915c_631e557f633f.slice/crio-4e876653205606ae3da6afb4e1f9126f931f42948c6ee22e5979bd09eee2ef39 WatchSource:0}: Error finding container 4e876653205606ae3da6afb4e1f9126f931f42948c6ee22e5979bd09eee2ef39: Status 404 returned error can't find the container with id 4e876653205606ae3da6afb4e1f9126f931f42948c6ee22e5979bd09eee2ef39 Jul 23 00:37:57.040820 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:57.040769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" event={"ID":"3fce81ce-1e9e-4430-915c-631e557f633f","Type":"ContainerStarted","Data":"4e876653205606ae3da6afb4e1f9126f931f42948c6ee22e5979bd09eee2ef39"} Jul 23 00:37:58.048390 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:58.048351 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" event={"ID":"3fce81ce-1e9e-4430-915c-631e557f633f","Type":"ContainerStarted","Data":"0d75d196e05aee60d23035d1a97c5299280fa739100960d9f830d20533b89fb8"} Jul 23 00:37:58.067243 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:58.067199 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" podStartSLOduration=1.85260347 podStartE2EDuration="3.067183089s" totalImagesPullingTime="1.214579619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:37:58.064801844 +0000 UTC m=+870.560470559" watchObservedRunningTime="2026-07-23 00:37:58.067183089 +0000 UTC m=+870.562851802" Jul 23 00:37:58.085346 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:37:58.085308 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:38:03.036499 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:38:03.036460 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:38:06.016629 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:38:06.016584 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:38:06.017045 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:38:06.016673 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:38:06.029222 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:38:06.029189 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:38:06.113847 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:38:06.113815 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9" Jul 23 00:38:12.192423 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:38:12.192381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:38:25.616687 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:38:25.616645 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:38:28.053609 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:38:28.053577 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:38:28.055103 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:38:28.055080 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:38:53.911444 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:38:53.911394 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:39:05.508952 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:05.508912 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:39:53.870170 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:53.870071 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68f94f797b-pghml"] Jul 23 00:39:53.873736 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:53.873714 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68f94f797b-pghml" Jul 23 00:39:53.881426 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:53.881397 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68f94f797b-pghml"] Jul 23 00:39:54.002094 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:54.002043 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7slw\" (UniqueName: \"kubernetes.io/projected/fb55a988-9291-44a6-b276-f26226fce713-kube-api-access-c7slw\") pod \"authorino-68f94f797b-pghml\" (UID: \"fb55a988-9291-44a6-b276-f26226fce713\") " pod="kuadrant-system/authorino-68f94f797b-pghml" Jul 23 00:39:54.002094 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:54.002090 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb55a988-9291-44a6-b276-f26226fce713-tls-cert\") pod \"authorino-68f94f797b-pghml\" (UID: \"fb55a988-9291-44a6-b276-f26226fce713\") " pod="kuadrant-system/authorino-68f94f797b-pghml" Jul 23 00:39:54.103618 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:54.103561 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7slw\" (UniqueName: \"kubernetes.io/projected/fb55a988-9291-44a6-b276-f26226fce713-kube-api-access-c7slw\") pod \"authorino-68f94f797b-pghml\" (UID: \"fb55a988-9291-44a6-b276-f26226fce713\") " pod="kuadrant-system/authorino-68f94f797b-pghml" Jul 23 00:39:54.103618 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:54.103608 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb55a988-9291-44a6-b276-f26226fce713-tls-cert\") pod \"authorino-68f94f797b-pghml\" (UID: \"fb55a988-9291-44a6-b276-f26226fce713\") " pod="kuadrant-system/authorino-68f94f797b-pghml" Jul 23 00:39:54.106184 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:54.106154 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb55a988-9291-44a6-b276-f26226fce713-tls-cert\") pod \"authorino-68f94f797b-pghml\" (UID: \"fb55a988-9291-44a6-b276-f26226fce713\") " pod="kuadrant-system/authorino-68f94f797b-pghml" Jul 23 00:39:54.112133 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:54.112098 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7slw\" (UniqueName: \"kubernetes.io/projected/fb55a988-9291-44a6-b276-f26226fce713-kube-api-access-c7slw\") pod \"authorino-68f94f797b-pghml\" (UID: \"fb55a988-9291-44a6-b276-f26226fce713\") " pod="kuadrant-system/authorino-68f94f797b-pghml" Jul 23 00:39:54.185967 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:54.185936 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68f94f797b-pghml" Jul 23 00:39:54.332582 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:54.332551 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68f94f797b-pghml"] Jul 23 00:39:54.333936 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:39:54.333910 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb55a988_9291_44a6_b276_f26226fce713.slice/crio-a9d0badf05053add8328da8b6c1b4fd6455ced5ef461abe334ceee0bbde3e765 WatchSource:0}: Error finding container a9d0badf05053add8328da8b6c1b4fd6455ced5ef461abe334ceee0bbde3e765: Status 404 returned error can't find the container with id a9d0badf05053add8328da8b6c1b4fd6455ced5ef461abe334ceee0bbde3e765 Jul 23 00:39:54.335397 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:54.335378 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:39:54.715395 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:54.715359 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68f94f797b-pghml" event={"ID":"fb55a988-9291-44a6-b276-f26226fce713","Type":"ContainerStarted","Data":"a9d0badf05053add8328da8b6c1b4fd6455ced5ef461abe334ceee0bbde3e765"} Jul 23 00:39:55.725143 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:55.725089 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68f94f797b-pghml" event={"ID":"fb55a988-9291-44a6-b276-f26226fce713","Type":"ContainerStarted","Data":"c21f2ba3987940e935d2abdadc5dc323ac1cdefec1768fa8eb9282017df84ce1"} Jul 23 00:39:55.748784 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:55.748710 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-68f94f797b-pghml" podStartSLOduration=2.324749422 podStartE2EDuration="2.748687339s" totalImagesPullingTime="423.937917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:39:55.746392937 +0000 UTC m=+988.242061651" watchObservedRunningTime="2026-07-23 00:39:55.748687339 +0000 UTC m=+988.244356052" Jul 23 00:39:55.780663 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:55.780618 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-797974f89-whc5d"] Jul 23 00:39:55.780904 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:55.780865 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-797974f89-whc5d" podUID="6b1c6037-ff7a-4484-86f4-c3a90813bd18" containerName="authorino" containerID="cri-o://6e6d1027baef46f0f5279b51e30f5db23329c9fd2f4fcfa76f459996e3c1bb79" gracePeriod=30 Jul 23 00:39:56.039178 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.039147 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-797974f89-whc5d" Jul 23 00:39:56.125080 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.125045 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2wsb\" (UniqueName: \"kubernetes.io/projected/6b1c6037-ff7a-4484-86f4-c3a90813bd18-kube-api-access-r2wsb\") pod \"6b1c6037-ff7a-4484-86f4-c3a90813bd18\" (UID: \"6b1c6037-ff7a-4484-86f4-c3a90813bd18\") " Jul 23 00:39:56.125260 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.125149 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6b1c6037-ff7a-4484-86f4-c3a90813bd18-tls-cert\") pod \"6b1c6037-ff7a-4484-86f4-c3a90813bd18\" (UID: \"6b1c6037-ff7a-4484-86f4-c3a90813bd18\") " Jul 23 00:39:56.127362 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.127316 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1c6037-ff7a-4484-86f4-c3a90813bd18-kube-api-access-r2wsb" (OuterVolumeSpecName: "kube-api-access-r2wsb") pod "6b1c6037-ff7a-4484-86f4-c3a90813bd18" (UID: "6b1c6037-ff7a-4484-86f4-c3a90813bd18"). InnerVolumeSpecName "kube-api-access-r2wsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:39:56.138223 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.138178 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b1c6037-ff7a-4484-86f4-c3a90813bd18-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6b1c6037-ff7a-4484-86f4-c3a90813bd18" (UID: "6b1c6037-ff7a-4484-86f4-c3a90813bd18"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:39:56.226612 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.226576 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6b1c6037-ff7a-4484-86f4-c3a90813bd18-tls-cert\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:39:56.226612 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.226609 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2wsb\" (UniqueName: \"kubernetes.io/projected/6b1c6037-ff7a-4484-86f4-c3a90813bd18-kube-api-access-r2wsb\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:39:56.307908 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.307807 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:39:56.404793 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.404752 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:39:56.733880 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.733838 2564 generic.go:356] "Generic (PLEG): container finished" podID="6b1c6037-ff7a-4484-86f4-c3a90813bd18" containerID="6e6d1027baef46f0f5279b51e30f5db23329c9fd2f4fcfa76f459996e3c1bb79" exitCode=0 Jul 23 00:39:56.733880 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.733892 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-797974f89-whc5d" Jul 23 00:39:56.734436 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.733929 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-797974f89-whc5d" event={"ID":"6b1c6037-ff7a-4484-86f4-c3a90813bd18","Type":"ContainerDied","Data":"6e6d1027baef46f0f5279b51e30f5db23329c9fd2f4fcfa76f459996e3c1bb79"} Jul 23 00:39:56.734436 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.733975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-797974f89-whc5d" event={"ID":"6b1c6037-ff7a-4484-86f4-c3a90813bd18","Type":"ContainerDied","Data":"8426c608fbec18c5b853ea31c3aa79fccc22cc9df6dacd1360d2d9c40cc0c7a6"} Jul 23 00:39:56.734436 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.733996 2564 scope.go:117] "RemoveContainer" containerID="6e6d1027baef46f0f5279b51e30f5db23329c9fd2f4fcfa76f459996e3c1bb79" Jul 23 00:39:56.745095 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.745068 2564 scope.go:117] "RemoveContainer" containerID="6e6d1027baef46f0f5279b51e30f5db23329c9fd2f4fcfa76f459996e3c1bb79" Jul 23 00:39:56.745399 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:39:56.745376 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6d1027baef46f0f5279b51e30f5db23329c9fd2f4fcfa76f459996e3c1bb79\": container with ID starting with 6e6d1027baef46f0f5279b51e30f5db23329c9fd2f4fcfa76f459996e3c1bb79 not found: ID does not exist" containerID="6e6d1027baef46f0f5279b51e30f5db23329c9fd2f4fcfa76f459996e3c1bb79" Jul 23 00:39:56.745470 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.745407 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6d1027baef46f0f5279b51e30f5db23329c9fd2f4fcfa76f459996e3c1bb79"} err="failed to get container status \"6e6d1027baef46f0f5279b51e30f5db23329c9fd2f4fcfa76f459996e3c1bb79\": rpc error: code = NotFound desc = could not find container \"6e6d1027baef46f0f5279b51e30f5db23329c9fd2f4fcfa76f459996e3c1bb79\": container with ID starting with 6e6d1027baef46f0f5279b51e30f5db23329c9fd2f4fcfa76f459996e3c1bb79 not found: ID does not exist" Jul 23 00:39:56.760636 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.760599 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-797974f89-whc5d"] Jul 23 00:39:56.764396 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:56.764360 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-797974f89-whc5d"] Jul 23 00:39:57.423620 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:57.423577 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:39:57.521099 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:57.521067 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:39:58.073140 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:39:58.073105 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b1c6037-ff7a-4484-86f4-c3a90813bd18" path="/var/lib/kubelet/pods/6b1c6037-ff7a-4484-86f4-c3a90813bd18/volumes" Jul 23 00:40:17.641228 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:40:17.641181 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:41:11.991265 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:41:11.991225 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:03.665140 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:42:03.665104 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:04.890051 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:42:04.890003 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:45.436824 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:42:45.436788 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:45.493867 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:42:45.493833 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:45.669725 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:42:45.669688 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:45.768178 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:42:45.768090 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:45.891987 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:42:45.891951 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:51.588683 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:42:51.588590 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:51.698835 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:42:51.698793 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:51.794151 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:42:51.794120 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:51.954920 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:42:51.954884 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:57.844921 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:42:57.844880 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:42:57.909752 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:42:57.909715 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:43:28.105478 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:43:28.105448 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:43:28.109866 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:43:28.109842 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:45:06.733008 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:45:06.732970 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:45:09.107276 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:45:09.107240 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:02.051597 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:47:02.051548 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:47:14.495297 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:47:14.495254 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:28.147761 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:48:28.147714 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:48:28.152008 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:48:28.151984 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:48:51.852210 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:48:51.852127 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:48:51.908572 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:48:51.908515 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:49:06.193750 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:49:06.193703 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:32.998667 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:50:32.998616 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:50:40.495969 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:50:40.495930 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:18.179946 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:51:18.179911 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:51:19.096130 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:51:19.096090 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:53:28.187904 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:53:28.187878 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:53:28.191817 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:53:28.191795 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:54:30.063339 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:54:30.063301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:36.806663 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:54:36.806629 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:36.876359 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:54:36.876326 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:51.477273 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:54:51.477187 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:54.412580 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:54:54.412538 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:54.515658 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:54:54.515626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:54:54.863218 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:54:54.863128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:00.473197 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:00.473156 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:00.548626 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:00.548585 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:00.715725 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:00.715695 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:01.025520 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:01.025484 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:01.170408 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:01.170366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:07.060346 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:07.060304 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:07.143957 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:07.143915 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:07.216573 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:07.216517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:07.314122 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:07.314024 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:07.839931 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:07.839897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:07.941572 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:07.941517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:32.630545 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:32.630498 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d"] Jul 23 00:55:32.630966 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:32.630933 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b1c6037-ff7a-4484-86f4-c3a90813bd18" containerName="authorino" Jul 23 00:55:32.630966 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:32.630945 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b1c6037-ff7a-4484-86f4-c3a90813bd18" containerName="authorino" Jul 23 00:55:32.631041 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:32.631036 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b1c6037-ff7a-4484-86f4-c3a90813bd18" containerName="authorino" Jul 23 00:55:32.634246 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:32.634224 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d" Jul 23 00:55:32.636969 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:32.636938 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:55:32.637147 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:32.637039 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zd6c\"" Jul 23 00:55:32.642056 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:32.642018 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d"] Jul 23 00:55:32.721975 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:32.721924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe92fa1-4c26-40f2-9e44-4556734c9ed9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5327912a-2nx9d\" (UID: \"7fe92fa1-4c26-40f2-9e44-4556734c9ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d" Jul 23 00:55:32.822742 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:32.822697 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe92fa1-4c26-40f2-9e44-4556734c9ed9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5327912a-2nx9d\" (UID: \"7fe92fa1-4c26-40f2-9e44-4556734c9ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d" Jul 23 00:55:32.823350 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:32.823326 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe92fa1-4c26-40f2-9e44-4556734c9ed9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5327912a-2nx9d\" (UID: \"7fe92fa1-4c26-40f2-9e44-4556734c9ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d" Jul 23 00:55:32.947357 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:32.947309 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d" Jul 23 00:55:33.097505 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:33.097472 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d"] Jul 23 00:55:33.099468 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:55:33.099432 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe92fa1_4c26_40f2_9e44_4556734c9ed9.slice/crio-0c10e4369dabc692bb1a6c067cd3598e7a2223ce29bc1a2e744b6d4edf98bd37 WatchSource:0}: Error finding container 0c10e4369dabc692bb1a6c067cd3598e7a2223ce29bc1a2e744b6d4edf98bd37: Status 404 returned error can't find the container with id 0c10e4369dabc692bb1a6c067cd3598e7a2223ce29bc1a2e744b6d4edf98bd37 Jul 23 00:55:33.101941 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:33.101916 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 00:55:34.092930 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:34.092889 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d" event={"ID":"7fe92fa1-4c26-40f2-9e44-4556734c9ed9","Type":"ContainerStarted","Data":"0c10e4369dabc692bb1a6c067cd3598e7a2223ce29bc1a2e744b6d4edf98bd37"} Jul 23 00:55:36.109744 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:36.109653 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d" event={"ID":"7fe92fa1-4c26-40f2-9e44-4556734c9ed9","Type":"ContainerStarted","Data":"8bff889211ebcf0411418b1c2b97b7adeb56951188b7144fa7ff55b0b784dc1c"} Jul 23 00:55:36.125456 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:36.125408 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d" podStartSLOduration=1.74115292 podStartE2EDuration="4.125392019s" totalImagesPullingTime="2.384239099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:55:36.12494278 +0000 UTC m=+1928.620611494" watchObservedRunningTime="2026-07-23 00:55:36.125392019 +0000 UTC m=+1928.621060734" Jul 23 00:55:37.118038 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:37.118001 2564 generic.go:356] "Generic (PLEG): container finished" podID="7fe92fa1-4c26-40f2-9e44-4556734c9ed9" containerID="8bff889211ebcf0411418b1c2b97b7adeb56951188b7144fa7ff55b0b784dc1c" exitCode=0 Jul 23 00:55:37.118487 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:37.118054 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d" event={"ID":"7fe92fa1-4c26-40f2-9e44-4556734c9ed9","Type":"ContainerDied","Data":"8bff889211ebcf0411418b1c2b97b7adeb56951188b7144fa7ff55b0b784dc1c"} Jul 23 00:55:38.267709 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:38.267668 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d" Jul 23 00:55:38.381006 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:38.380894 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe92fa1-4c26-40f2-9e44-4556734c9ed9-maas-api-service-ca\") pod \"7fe92fa1-4c26-40f2-9e44-4556734c9ed9\" (UID: \"7fe92fa1-4c26-40f2-9e44-4556734c9ed9\") " Jul 23 00:55:38.381296 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:38.381270 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe92fa1-4c26-40f2-9e44-4556734c9ed9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7fe92fa1-4c26-40f2-9e44-4556734c9ed9" (UID: "7fe92fa1-4c26-40f2-9e44-4556734c9ed9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:55:38.482303 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:38.482259 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe92fa1-4c26-40f2-9e44-4556734c9ed9-maas-api-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:55:39.131212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:39.131176 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d" Jul 23 00:55:39.131212 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:39.131201 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d" event={"ID":"7fe92fa1-4c26-40f2-9e44-4556734c9ed9","Type":"ContainerDied","Data":"0c10e4369dabc692bb1a6c067cd3598e7a2223ce29bc1a2e744b6d4edf98bd37"} Jul 23 00:55:39.131517 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:39.131251 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c10e4369dabc692bb1a6c067cd3598e7a2223ce29bc1a2e744b6d4edf98bd37" Jul 23 00:55:43.097935 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:43.097897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:47.708797 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:47.708761 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:47.710321 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:47.710296 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d"] Jul 23 00:55:47.713994 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:47.713954 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5327912a-2nx9d"] Jul 23 00:55:47.863552 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:47.863505 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:48.073506 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:48.073413 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fe92fa1-4c26-40f2-9e44-4556734c9ed9" path="/var/lib/kubelet/pods/7fe92fa1-4c26-40f2-9e44-4556734c9ed9/volumes" Jul 23 00:55:48.111516 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:48.111483 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:48.377831 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:48.377731 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:48.479581 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:48.479521 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:48.592485 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:48.592448 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:48.896017 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:48.895986 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:49.174114 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:49.174000 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:51.476400 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:51.476344 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:51.676024 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:51.675988 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:51.777320 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:51.777190 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:52.226261 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:52.226229 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:55:52.328049 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:55:52.328001 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:16.405930 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:16.405884 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78"] Jul 23 00:56:16.406411 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:16.406305 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe92fa1-4c26-40f2-9e44-4556734c9ed9" containerName="revoke-keys" Jul 23 00:56:16.406411 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:16.406316 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe92fa1-4c26-40f2-9e44-4556734c9ed9" containerName="revoke-keys" Jul 23 00:56:16.406411 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:16.406392 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe92fa1-4c26-40f2-9e44-4556734c9ed9" containerName="revoke-keys" Jul 23 00:56:16.413295 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:16.413267 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78" Jul 23 00:56:16.416125 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:16.416094 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zd6c\"" Jul 23 00:56:16.416296 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:16.416094 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:56:16.416821 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:16.416793 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78"] Jul 23 00:56:16.468103 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:16.468063 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78\" (UID: \"aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78" Jul 23 00:56:16.569764 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:16.569719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78\" (UID: \"aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78" Jul 23 00:56:16.570383 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:16.570360 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78\" (UID: \"aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78" Jul 23 00:56:16.725624 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:16.725512 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78" Jul 23 00:56:16.871029 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:16.870999 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78"] Jul 23 00:56:16.873001 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:56:16.872967 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff8a9c7_2a1b_4dcb_8bf6_cb07e231fc56.slice/crio-c89ae6a7deba7356269a9672ad589f69c5b55f298f97173ebe9e85c8b1c8fe13 WatchSource:0}: Error finding container c89ae6a7deba7356269a9672ad589f69c5b55f298f97173ebe9e85c8b1c8fe13: Status 404 returned error can't find the container with id c89ae6a7deba7356269a9672ad589f69c5b55f298f97173ebe9e85c8b1c8fe13 Jul 23 00:56:17.360132 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:17.360028 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78" event={"ID":"aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56","Type":"ContainerStarted","Data":"569c4111b50d1cec54872a857bc4e5652cc20be6293eba418a709963790bfa9b"} Jul 23 00:56:17.360132 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:17.360090 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78" event={"ID":"aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56","Type":"ContainerStarted","Data":"c89ae6a7deba7356269a9672ad589f69c5b55f298f97173ebe9e85c8b1c8fe13"} Jul 23 00:56:17.375546 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:17.375467 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78" podStartSLOduration=1.3754472739999999 podStartE2EDuration="1.375447274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:56:17.37404575 +0000 UTC m=+1969.869714463" watchObservedRunningTime="2026-07-23 00:56:17.375447274 +0000 UTC m=+1969.871115987" Jul 23 00:56:18.374652 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:18.374616 2564 generic.go:356] "Generic (PLEG): container finished" podID="aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56" containerID="569c4111b50d1cec54872a857bc4e5652cc20be6293eba418a709963790bfa9b" exitCode=0 Jul 23 00:56:18.375056 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:18.374709 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78" event={"ID":"aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56","Type":"ContainerDied","Data":"569c4111b50d1cec54872a857bc4e5652cc20be6293eba418a709963790bfa9b"} Jul 23 00:56:19.527381 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:19.527344 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78" Jul 23 00:56:19.599495 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:19.599434 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56-maas-api-service-ca\") pod \"aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56\" (UID: \"aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56\") " Jul 23 00:56:19.599846 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:19.599823 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56" (UID: "aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:56:19.701148 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:19.701102 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56-maas-api-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:56:20.388347 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:20.388311 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78" Jul 23 00:56:20.388522 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:20.388319 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78" event={"ID":"aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56","Type":"ContainerDied","Data":"c89ae6a7deba7356269a9672ad589f69c5b55f298f97173ebe9e85c8b1c8fe13"} Jul 23 00:56:20.388522 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:20.388441 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c89ae6a7deba7356269a9672ad589f69c5b55f298f97173ebe9e85c8b1c8fe13" Jul 23 00:56:26.895653 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:26.895612 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:31.485685 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:31.485634 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:31.486829 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:31.486805 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78"] Jul 23 00:56:31.489302 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:31.489277 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-5d1adec3-fqq78"] Jul 23 00:56:31.644643 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:31.644611 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:31.875619 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:31.875509 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:32.073165 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:32.073118 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56" path="/var/lib/kubelet/pods/aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56/volumes" Jul 23 00:56:32.097247 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:32.097214 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:32.429626 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:32.429576 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:32.864442 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:32.864350 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:34.533245 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.533203 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:34.655581 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.655523 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42"] Jul 23 00:56:34.656005 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.655990 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56" containerName="revoke-keys" Jul 23 00:56:34.656060 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.656007 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56" containerName="revoke-keys" Jul 23 00:56:34.656099 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.656071 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff8a9c7-2a1b-4dcb-8bf6-cb07e231fc56" containerName="revoke-keys" Jul 23 00:56:34.659244 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.659214 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:56:34.662322 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.662290 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-882c377a-serving-cert\"" Jul 23 00:56:34.662574 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.662554 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sq2ww\"" Jul 23 00:56:34.663436 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.663416 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:56:34.664010 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.663991 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 00:56:34.683042 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.683002 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:34.685909 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.685875 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42"] Jul 23 00:56:34.766502 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.766467 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:34.855561 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.855439 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-openshift-service-ca\") pod \"maas-api-e2e-ait-882c377a-557466f8bc-w9n42\" (UID: \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:56:34.855738 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.855639 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-maas-api-tls\") pod \"maas-api-e2e-ait-882c377a-557466f8bc-w9n42\" (UID: \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:56:34.855738 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.855667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv5jk\" (UniqueName: \"kubernetes.io/projected/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-kube-api-access-sv5jk\") pod \"maas-api-e2e-ait-882c377a-557466f8bc-w9n42\" (UID: \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:56:34.956446 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.956405 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-maas-api-tls\") pod \"maas-api-e2e-ait-882c377a-557466f8bc-w9n42\" (UID: \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:56:34.956446 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.956447 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv5jk\" (UniqueName: \"kubernetes.io/projected/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-kube-api-access-sv5jk\") pod \"maas-api-e2e-ait-882c377a-557466f8bc-w9n42\" (UID: \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:56:34.956709 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.956503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-openshift-service-ca\") pod \"maas-api-e2e-ait-882c377a-557466f8bc-w9n42\" (UID: \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:56:34.956709 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:56:34.956593 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-serving-cert: secret "maas-api-e2e-ait-882c377a-serving-cert" not found Jul 23 00:56:34.956709 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:56:34.956666 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-maas-api-tls podName:ff69c8fd-f667-4ac8-8d82-e9534d2590b0 nodeName:}" failed. No retries permitted until 2026-07-23 00:56:35.456644357 +0000 UTC m=+1987.952313049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-maas-api-tls") pod "maas-api-e2e-ait-882c377a-557466f8bc-w9n42" (UID: "ff69c8fd-f667-4ac8-8d82-e9534d2590b0") : secret "maas-api-e2e-ait-882c377a-serving-cert" not found Jul 23 00:56:34.957256 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.957238 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-openshift-service-ca\") pod \"maas-api-e2e-ait-882c377a-557466f8bc-w9n42\" (UID: \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:56:34.968474 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:34.968441 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv5jk\" (UniqueName: \"kubernetes.io/projected/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-kube-api-access-sv5jk\") pod \"maas-api-e2e-ait-882c377a-557466f8bc-w9n42\" (UID: \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:56:35.196274 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:35.196236 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:35.297880 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:35.297843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:35.400159 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:35.400118 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:56:35.462665 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:35.462551 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-maas-api-tls\") pod \"maas-api-e2e-ait-882c377a-557466f8bc-w9n42\" (UID: \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:56:35.465225 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:35.465194 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-maas-api-tls\") pod \"maas-api-e2e-ait-882c377a-557466f8bc-w9n42\" (UID: \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:56:35.571325 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:35.571289 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:56:35.720352 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:35.720319 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42"] Jul 23 00:56:35.723258 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:56:35.723223 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff69c8fd_f667_4ac8_8d82_e9534d2590b0.slice/crio-c0e4e67e2f0713128dfc3495f09fb369a4dfd43fba9d8ca15f666f9531972cd4 WatchSource:0}: Error finding container c0e4e67e2f0713128dfc3495f09fb369a4dfd43fba9d8ca15f666f9531972cd4: Status 404 returned error can't find the container with id c0e4e67e2f0713128dfc3495f09fb369a4dfd43fba9d8ca15f666f9531972cd4 Jul 23 00:56:36.490693 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:36.490648 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" event={"ID":"ff69c8fd-f667-4ac8-8d82-e9534d2590b0","Type":"ContainerStarted","Data":"c0e4e67e2f0713128dfc3495f09fb369a4dfd43fba9d8ca15f666f9531972cd4"} Jul 23 00:56:38.506952 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:38.506901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" event={"ID":"ff69c8fd-f667-4ac8-8d82-e9534d2590b0","Type":"ContainerStarted","Data":"d4d92397a89050fc5e6b4f59dc08aeb4dbb86f9415817e9f8e7468619cf6a26f"} Jul 23 00:56:38.507456 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:38.507007 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:56:38.526548 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:38.526455 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" podStartSLOduration=2.279173706 podStartE2EDuration="4.526429937s" totalImagesPullingTime="2.247256231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:56:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:56:38.523679354 +0000 UTC m=+1991.019348069" watchObservedRunningTime="2026-07-23 00:56:38.526429937 +0000 UTC m=+1991.022098652" Jul 23 00:56:44.518896 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:44.518865 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:56:44.813322 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:56:44.813232 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:10.842223 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:10.842182 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42"] Jul 23 00:57:10.842685 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:10.842462 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" podUID="ff69c8fd-f667-4ac8-8d82-e9534d2590b0" containerName="maas-api" containerID="cri-o://d4d92397a89050fc5e6b4f59dc08aeb4dbb86f9415817e9f8e7468619cf6a26f" gracePeriod=30 Jul 23 00:57:10.868758 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:10.868711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:11.101402 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.101328 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:57:11.127641 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.127610 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-openshift-service-ca\") pod \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\" (UID: \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\") " Jul 23 00:57:11.127837 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.127675 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-maas-api-tls\") pod \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\" (UID: \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\") " Jul 23 00:57:11.127837 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.127781 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv5jk\" (UniqueName: \"kubernetes.io/projected/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-kube-api-access-sv5jk\") pod \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\" (UID: \"ff69c8fd-f667-4ac8-8d82-e9534d2590b0\") " Jul 23 00:57:11.128022 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.127991 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ff69c8fd-f667-4ac8-8d82-e9534d2590b0" (UID: "ff69c8fd-f667-4ac8-8d82-e9534d2590b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:57:11.130025 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.129987 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-kube-api-access-sv5jk" (OuterVolumeSpecName: "kube-api-access-sv5jk") pod "ff69c8fd-f667-4ac8-8d82-e9534d2590b0" (UID: "ff69c8fd-f667-4ac8-8d82-e9534d2590b0"). InnerVolumeSpecName "kube-api-access-sv5jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:57:11.130025 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.130013 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ff69c8fd-f667-4ac8-8d82-e9534d2590b0" (UID: "ff69c8fd-f667-4ac8-8d82-e9534d2590b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:57:11.228922 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.228877 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv5jk\" (UniqueName: \"kubernetes.io/projected/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-kube-api-access-sv5jk\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:57:11.228922 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.228913 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-openshift-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:57:11.228922 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.228926 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff69c8fd-f667-4ac8-8d82-e9534d2590b0-maas-api-tls\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:57:11.701392 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.701358 2564 generic.go:356] "Generic (PLEG): container finished" podID="ff69c8fd-f667-4ac8-8d82-e9534d2590b0" containerID="d4d92397a89050fc5e6b4f59dc08aeb4dbb86f9415817e9f8e7468619cf6a26f" exitCode=0 Jul 23 00:57:11.701595 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.701407 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" Jul 23 00:57:11.701595 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.701441 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" event={"ID":"ff69c8fd-f667-4ac8-8d82-e9534d2590b0","Type":"ContainerDied","Data":"d4d92397a89050fc5e6b4f59dc08aeb4dbb86f9415817e9f8e7468619cf6a26f"} Jul 23 00:57:11.701595 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.701493 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42" event={"ID":"ff69c8fd-f667-4ac8-8d82-e9534d2590b0","Type":"ContainerDied","Data":"c0e4e67e2f0713128dfc3495f09fb369a4dfd43fba9d8ca15f666f9531972cd4"} Jul 23 00:57:11.701595 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.701516 2564 scope.go:117] "RemoveContainer" containerID="d4d92397a89050fc5e6b4f59dc08aeb4dbb86f9415817e9f8e7468619cf6a26f" Jul 23 00:57:11.712890 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.712870 2564 scope.go:117] "RemoveContainer" containerID="d4d92397a89050fc5e6b4f59dc08aeb4dbb86f9415817e9f8e7468619cf6a26f" Jul 23 00:57:11.713210 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:57:11.713188 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d92397a89050fc5e6b4f59dc08aeb4dbb86f9415817e9f8e7468619cf6a26f\": container with ID starting with d4d92397a89050fc5e6b4f59dc08aeb4dbb86f9415817e9f8e7468619cf6a26f not found: ID does not exist" containerID="d4d92397a89050fc5e6b4f59dc08aeb4dbb86f9415817e9f8e7468619cf6a26f" Jul 23 00:57:11.713288 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.713219 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d92397a89050fc5e6b4f59dc08aeb4dbb86f9415817e9f8e7468619cf6a26f"} err="failed to get container status \"d4d92397a89050fc5e6b4f59dc08aeb4dbb86f9415817e9f8e7468619cf6a26f\": rpc error: code = NotFound desc = could not find container \"d4d92397a89050fc5e6b4f59dc08aeb4dbb86f9415817e9f8e7468619cf6a26f\": container with ID starting with d4d92397a89050fc5e6b4f59dc08aeb4dbb86f9415817e9f8e7468619cf6a26f not found: ID does not exist" Jul 23 00:57:11.726841 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.726795 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42"] Jul 23 00:57:11.730979 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:11.730941 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-882c377a-557466f8bc-w9n42"] Jul 23 00:57:12.073375 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:12.073281 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff69c8fd-f667-4ac8-8d82-e9534d2590b0" path="/var/lib/kubelet/pods/ff69c8fd-f667-4ac8-8d82-e9534d2590b0/volumes" Jul 23 00:57:15.386231 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:15.386199 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:16.601363 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:16.601326 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:16.882996 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:16.882908 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:17.077054 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:17.077020 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:17.385585 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:17.385551 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:17.461489 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:17.461458 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:19.718997 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.718948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:19.797873 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.797826 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:19.798628 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.798508 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2"] Jul 23 00:57:19.799259 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.799237 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff69c8fd-f667-4ac8-8d82-e9534d2590b0" containerName="maas-api" Jul 23 00:57:19.799341 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.799263 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff69c8fd-f667-4ac8-8d82-e9534d2590b0" containerName="maas-api" Jul 23 00:57:19.799413 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.799398 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff69c8fd-f667-4ac8-8d82-e9534d2590b0" containerName="maas-api" Jul 23 00:57:19.804204 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.804182 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:19.806930 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.806908 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-cc4a0bee-serving-cert\"" Jul 23 00:57:19.807198 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.807183 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 00:57:19.808356 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.808333 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:57:19.808589 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.808571 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sq2ww\"" Jul 23 00:57:19.813055 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.812556 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2"] Jul 23 00:57:19.878614 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.878579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:19.908076 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.908048 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpzlp\" (UniqueName: \"kubernetes.io/projected/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-kube-api-access-zpzlp\") pod \"maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2\" (UID: \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:19.908242 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.908184 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-maas-api-tls\") pod \"maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2\" (UID: \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:19.908242 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:19.908230 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-openshift-service-ca\") pod \"maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2\" (UID: \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:20.009574 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:20.009494 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpzlp\" (UniqueName: \"kubernetes.io/projected/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-kube-api-access-zpzlp\") pod \"maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2\" (UID: \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:20.009709 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:20.009579 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-maas-api-tls\") pod \"maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2\" (UID: \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:20.009709 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:20.009603 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-openshift-service-ca\") pod \"maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2\" (UID: \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:20.009820 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:57:20.009728 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-serving-cert: secret "maas-api-e2e-derive-cc4a0bee-serving-cert" not found Jul 23 00:57:20.009820 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:57:20.009806 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-maas-api-tls podName:5808a8aa-41d2-4ee0-83bc-ec9e6201ea04 nodeName:}" failed. No retries permitted until 2026-07-23 00:57:20.509784539 +0000 UTC m=+2033.005453234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-maas-api-tls") pod "maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" (UID: "5808a8aa-41d2-4ee0-83bc-ec9e6201ea04") : secret "maas-api-e2e-derive-cc4a0bee-serving-cert" not found Jul 23 00:57:20.010193 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:20.010176 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-openshift-service-ca\") pod \"maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2\" (UID: \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:20.030651 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:20.030631 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpzlp\" (UniqueName: \"kubernetes.io/projected/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-kube-api-access-zpzlp\") pod \"maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2\" (UID: \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:20.165008 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:20.164979 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:20.514480 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:20.514448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-maas-api-tls\") pod \"maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2\" (UID: \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:20.516954 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:20.516929 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-maas-api-tls\") pod \"maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2\" (UID: \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:20.615381 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:20.615337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:20.717773 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:20.717747 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:20.720368 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:20.720337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:20.868173 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:20.868150 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2"] Jul 23 00:57:20.869177 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:57:20.869148 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5808a8aa_41d2_4ee0_83bc_ec9e6201ea04.slice/crio-14d7d9b6a58f56f81e61e0aa0721345d78d7b4805692988f5f4d1389f720a22d WatchSource:0}: Error finding container 14d7d9b6a58f56f81e61e0aa0721345d78d7b4805692988f5f4d1389f720a22d: Status 404 returned error can't find the container with id 14d7d9b6a58f56f81e61e0aa0721345d78d7b4805692988f5f4d1389f720a22d Jul 23 00:57:21.771509 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:21.771472 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" event={"ID":"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04","Type":"ContainerStarted","Data":"0d84531ff856db896a89ab377fc547d6b6b99ede38506117914f2337113714ec"} Jul 23 00:57:21.771919 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:21.771550 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" event={"ID":"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04","Type":"ContainerStarted","Data":"14d7d9b6a58f56f81e61e0aa0721345d78d7b4805692988f5f4d1389f720a22d"} Jul 23 00:57:21.771919 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:21.771693 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:21.793674 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:21.793635 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" podStartSLOduration=2.447727043 podStartE2EDuration="2.793619376s" totalImagesPullingTime="345.892333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:57:21.792143504 +0000 UTC m=+2034.287812234" watchObservedRunningTime="2026-07-23 00:57:21.793619376 +0000 UTC m=+2034.289288089" Jul 23 00:57:26.743800 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:26.743752 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:26.843902 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:26.843869 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:27.782129 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:27.782097 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:42.549522 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:42.549489 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:42.666783 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:42.666755 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:55.016245 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.016055 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2"] Jul 23 00:57:55.017405 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.017098 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" podUID="5808a8aa-41d2-4ee0-83bc-ec9e6201ea04" containerName="maas-api" containerID="cri-o://0d84531ff856db896a89ab377fc547d6b6b99ede38506117914f2337113714ec" gracePeriod=30 Jul 23 00:57:55.030854 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.030832 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:55.274096 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.274039 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:55.312776 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.312748 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-openshift-service-ca\") pod \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\" (UID: \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\") " Jul 23 00:57:55.312908 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.312792 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-maas-api-tls\") pod \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\" (UID: \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\") " Jul 23 00:57:55.312908 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.312822 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpzlp\" (UniqueName: \"kubernetes.io/projected/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-kube-api-access-zpzlp\") pod \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\" (UID: \"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04\") " Jul 23 00:57:55.313064 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.313042 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5808a8aa-41d2-4ee0-83bc-ec9e6201ea04" (UID: "5808a8aa-41d2-4ee0-83bc-ec9e6201ea04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:57:55.314959 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.314930 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-kube-api-access-zpzlp" (OuterVolumeSpecName: "kube-api-access-zpzlp") pod "5808a8aa-41d2-4ee0-83bc-ec9e6201ea04" (UID: "5808a8aa-41d2-4ee0-83bc-ec9e6201ea04"). InnerVolumeSpecName "kube-api-access-zpzlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 00:57:55.315051 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.314977 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5808a8aa-41d2-4ee0-83bc-ec9e6201ea04" (UID: "5808a8aa-41d2-4ee0-83bc-ec9e6201ea04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 00:57:55.413428 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.413393 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-openshift-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:57:55.413428 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.413421 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-maas-api-tls\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:57:55.413428 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.413431 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpzlp\" (UniqueName: \"kubernetes.io/projected/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04-kube-api-access-zpzlp\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:57:55.961879 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.961847 2564 generic.go:356] "Generic (PLEG): container finished" podID="5808a8aa-41d2-4ee0-83bc-ec9e6201ea04" containerID="0d84531ff856db896a89ab377fc547d6b6b99ede38506117914f2337113714ec" exitCode=0 Jul 23 00:57:55.962078 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.961896 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" Jul 23 00:57:55.962078 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.961938 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" event={"ID":"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04","Type":"ContainerDied","Data":"0d84531ff856db896a89ab377fc547d6b6b99ede38506117914f2337113714ec"} Jul 23 00:57:55.962078 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.962002 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2" event={"ID":"5808a8aa-41d2-4ee0-83bc-ec9e6201ea04","Type":"ContainerDied","Data":"14d7d9b6a58f56f81e61e0aa0721345d78d7b4805692988f5f4d1389f720a22d"} Jul 23 00:57:55.962078 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.962030 2564 scope.go:117] "RemoveContainer" containerID="0d84531ff856db896a89ab377fc547d6b6b99ede38506117914f2337113714ec" Jul 23 00:57:55.978981 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.978961 2564 scope.go:117] "RemoveContainer" containerID="0d84531ff856db896a89ab377fc547d6b6b99ede38506117914f2337113714ec" Jul 23 00:57:55.979243 ip-10-0-139-1 kubenswrapper[2564]: E0723 00:57:55.979224 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d84531ff856db896a89ab377fc547d6b6b99ede38506117914f2337113714ec\": container with ID starting with 0d84531ff856db896a89ab377fc547d6b6b99ede38506117914f2337113714ec not found: ID does not exist" containerID="0d84531ff856db896a89ab377fc547d6b6b99ede38506117914f2337113714ec" Jul 23 00:57:55.979292 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.979251 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d84531ff856db896a89ab377fc547d6b6b99ede38506117914f2337113714ec"} err="failed to get container status \"0d84531ff856db896a89ab377fc547d6b6b99ede38506117914f2337113714ec\": rpc error: code = NotFound desc = could not find container \"0d84531ff856db896a89ab377fc547d6b6b99ede38506117914f2337113714ec\": container with ID starting with 0d84531ff856db896a89ab377fc547d6b6b99ede38506117914f2337113714ec not found: ID does not exist" Jul 23 00:57:55.988288 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.988265 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2"] Jul 23 00:57:55.991472 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:55.991452 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cc4a0bee-dbcd86b66-d8bb2"] Jul 23 00:57:56.072986 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:56.072964 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5808a8aa-41d2-4ee0-83bc-ec9e6201ea04" path="/var/lib/kubelet/pods/5808a8aa-41d2-4ee0-83bc-ec9e6201ea04/volumes" Jul 23 00:57:59.665651 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:59.665601 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:57:59.843959 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:57:59.843907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:00.085726 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:00.085628 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:00.562434 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:00.562400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:00.659875 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:00.659845 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:00.758584 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:00.758539 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:02.176239 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:02.176202 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:08.028287 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:08.028248 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:08.134714 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:08.134678 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:08.230085 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:08.230048 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:08.330640 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:08.330568 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:08.751584 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:08.751550 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:13.612302 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:13.612271 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:28.238865 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:28.238842 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:58:28.243027 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:28.243007 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 00:58:34.725072 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:34.725038 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:38.096077 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:38.096037 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:40.941867 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:40.941831 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:41.047702 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:41.047662 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:41.141580 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:41.141519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:58:57.044882 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:58:57.044832 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:03.037110 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:03.037069 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:03.105665 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:03.105636 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:03.184037 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:03.184005 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:03.279540 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:03.279506 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:13.765315 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:13.765255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:14.561328 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:14.561295 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:20.994773 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:20.994690 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:21.097418 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:21.097383 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:21.194867 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:21.194838 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:21.295190 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:21.295100 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:22.074011 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:22.073971 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:22.177537 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:22.177504 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:22.279613 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:22.279582 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:23.878519 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:23.878483 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:30.225207 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:30.225168 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:30.323173 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:30.323134 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:30.894700 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:30.894655 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:37.414446 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:37.414400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:37.512053 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:37.511960 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:37.657426 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:37.657367 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:37.980588 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:37.980546 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:38.495227 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:38.495191 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:41.060331 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:41.060295 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:41.166992 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:41.166949 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:41.296642 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:41.296606 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:41.428368 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:41.428328 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:41.861260 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:41.861174 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:41.961813 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:41.961779 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:45.457553 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:45.457502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:46.131113 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:46.131081 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:46.235373 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:46.235343 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:46.549408 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:46.549378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:52.142388 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:52.142353 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:52.242836 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:52.242804 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 00:59:52.509609 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:52.509514 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr"] Jul 23 00:59:52.509944 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:52.509930 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5808a8aa-41d2-4ee0-83bc-ec9e6201ea04" containerName="maas-api" Jul 23 00:59:52.509989 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:52.509945 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5808a8aa-41d2-4ee0-83bc-ec9e6201ea04" containerName="maas-api" Jul 23 00:59:52.510028 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:52.510014 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5808a8aa-41d2-4ee0-83bc-ec9e6201ea04" containerName="maas-api" Jul 23 00:59:52.513040 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:52.513024 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr" Jul 23 00:59:52.515946 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:52.515924 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 00:59:52.516073 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:52.516022 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zd6c\"" Jul 23 00:59:52.521434 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:52.521142 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr"] Jul 23 00:59:52.669672 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:52.669636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd4b182-ee4b-4208-afec-b0b3a63fbb77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr\" (UID: \"0bd4b182-ee4b-4208-afec-b0b3a63fbb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr" Jul 23 00:59:52.771047 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:52.770942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd4b182-ee4b-4208-afec-b0b3a63fbb77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr\" (UID: \"0bd4b182-ee4b-4208-afec-b0b3a63fbb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr" Jul 23 00:59:52.771611 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:52.771592 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd4b182-ee4b-4208-afec-b0b3a63fbb77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr\" (UID: \"0bd4b182-ee4b-4208-afec-b0b3a63fbb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr" Jul 23 00:59:52.826038 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:52.825999 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr" Jul 23 00:59:52.958179 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:52.958152 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr"] Jul 23 00:59:52.960008 ip-10-0-139-1 kubenswrapper[2564]: W0723 00:59:52.959980 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd4b182_ee4b_4208_afec_b0b3a63fbb77.slice/crio-2368b7bb746267394414314b375c9c1192a1180de844abe2c67cc20f3fb6815e WatchSource:0}: Error finding container 2368b7bb746267394414314b375c9c1192a1180de844abe2c67cc20f3fb6815e: Status 404 returned error can't find the container with id 2368b7bb746267394414314b375c9c1192a1180de844abe2c67cc20f3fb6815e Jul 23 00:59:53.623062 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:53.622958 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr" event={"ID":"0bd4b182-ee4b-4208-afec-b0b3a63fbb77","Type":"ContainerStarted","Data":"f9a86b5c7c2b775414ac657ea5450be104017c84b0fbb347ce7d59f9dcfeca77"} Jul 23 00:59:53.623062 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:53.623010 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr" event={"ID":"0bd4b182-ee4b-4208-afec-b0b3a63fbb77","Type":"ContainerStarted","Data":"2368b7bb746267394414314b375c9c1192a1180de844abe2c67cc20f3fb6815e"} Jul 23 00:59:53.638853 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:53.638793 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr" podStartSLOduration=1.638774588 podStartE2EDuration="1.638774588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 00:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 00:59:53.638601045 +0000 UTC m=+2186.134269758" watchObservedRunningTime="2026-07-23 00:59:53.638774588 +0000 UTC m=+2186.134443305" Jul 23 00:59:54.630560 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:54.630502 2564 generic.go:356] "Generic (PLEG): container finished" podID="0bd4b182-ee4b-4208-afec-b0b3a63fbb77" containerID="f9a86b5c7c2b775414ac657ea5450be104017c84b0fbb347ce7d59f9dcfeca77" exitCode=0 Jul 23 00:59:54.630963 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:54.630563 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr" event={"ID":"0bd4b182-ee4b-4208-afec-b0b3a63fbb77","Type":"ContainerDied","Data":"f9a86b5c7c2b775414ac657ea5450be104017c84b0fbb347ce7d59f9dcfeca77"} Jul 23 00:59:55.770066 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:55.770044 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr" Jul 23 00:59:55.900559 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:55.900463 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd4b182-ee4b-4208-afec-b0b3a63fbb77-maas-api-service-ca\") pod \"0bd4b182-ee4b-4208-afec-b0b3a63fbb77\" (UID: \"0bd4b182-ee4b-4208-afec-b0b3a63fbb77\") " Jul 23 00:59:55.900822 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:55.900799 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd4b182-ee4b-4208-afec-b0b3a63fbb77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0bd4b182-ee4b-4208-afec-b0b3a63fbb77" (UID: "0bd4b182-ee4b-4208-afec-b0b3a63fbb77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 00:59:56.001239 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:56.001211 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd4b182-ee4b-4208-afec-b0b3a63fbb77-maas-api-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 00:59:56.643698 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:56.643666 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr" event={"ID":"0bd4b182-ee4b-4208-afec-b0b3a63fbb77","Type":"ContainerDied","Data":"2368b7bb746267394414314b375c9c1192a1180de844abe2c67cc20f3fb6815e"} Jul 23 00:59:56.643698 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:56.643692 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr" Jul 23 00:59:56.643698 ip-10-0-139-1 kubenswrapper[2564]: I0723 00:59:56.643713 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2368b7bb746267394414314b375c9c1192a1180de844abe2c67cc20f3fb6815e" Jul 23 01:00:00.142279 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.142245 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746140-d7dq5"] Jul 23 01:00:00.142695 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.142674 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd4b182-ee4b-4208-afec-b0b3a63fbb77" containerName="revoke-keys" Jul 23 01:00:00.142695 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.142688 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd4b182-ee4b-4208-afec-b0b3a63fbb77" containerName="revoke-keys" Jul 23 01:00:00.142769 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.142759 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd4b182-ee4b-4208-afec-b0b3a63fbb77" containerName="revoke-keys" Jul 23 01:00:00.145815 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.145796 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-489ccc2d-29746140-clv4p"] Jul 23 01:00:00.145919 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.145904 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-d7dq5" Jul 23 01:00:00.149210 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.149192 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-489ccc2d-29746140-clv4p" Jul 23 01:00:00.149393 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.149378 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zd6c\"" Jul 23 01:00:00.154507 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.154483 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746140-d7dq5"] Jul 23 01:00:00.158306 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.158285 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-d7dq5" Jul 23 01:00:00.165055 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.165038 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-489ccc2d-29746140-clv4p" Jul 23 01:00:00.171437 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.171414 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-489ccc2d-29746140-clv4p"] Jul 23 01:00:00.312299 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.312266 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29746140-d7dq5"] Jul 23 01:00:00.312497 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:00:00.312470 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d3dadb_1ef5_4c10_915c_38152ce5d26c.slice/crio-9d45d4b63179275cb34a92ab4d4d500d5e285e353d2070108603b7ab04ab2e9f WatchSource:0}: Error finding container 9d45d4b63179275cb34a92ab4d4d500d5e285e353d2070108603b7ab04ab2e9f: Status 404 returned error can't find the container with id 9d45d4b63179275cb34a92ab4d4d500d5e285e353d2070108603b7ab04ab2e9f Jul 23 01:00:00.330069 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.330044 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-489ccc2d-29746140-clv4p"] Jul 23 01:00:00.330902 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:00:00.330881 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode020ab3f_88e3_4348_8368_6fad82386889.slice/crio-1582d584b24ece96c2a685ddd819fa7b15a926efbca55ada2a88e8637d8b9325 WatchSource:0}: Error finding container 1582d584b24ece96c2a685ddd819fa7b15a926efbca55ada2a88e8637d8b9325: Status 404 returned error can't find the container with id 1582d584b24ece96c2a685ddd819fa7b15a926efbca55ada2a88e8637d8b9325 Jul 23 01:00:00.666338 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.666310 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-489ccc2d-29746140-clv4p" event={"ID":"e020ab3f-88e3-4348-8368-6fad82386889","Type":"ContainerStarted","Data":"1582d584b24ece96c2a685ddd819fa7b15a926efbca55ada2a88e8637d8b9325"} Jul 23 01:00:00.667728 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.667703 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-d7dq5" event={"ID":"e5d3dadb-1ef5-4c10-915c-38152ce5d26c","Type":"ContainerStarted","Data":"3f06b4db2757a91cc4d16929309a8719d28ecbf724b1e3fd16d47240b39cc9e4"} Jul 23 01:00:00.667852 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.667754 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-d7dq5" event={"ID":"e5d3dadb-1ef5-4c10-915c-38152ce5d26c","Type":"ContainerStarted","Data":"9d45d4b63179275cb34a92ab4d4d500d5e285e353d2070108603b7ab04ab2e9f"} Jul 23 01:00:00.683760 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:00.683720 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-d7dq5" podStartSLOduration=0.683704361 podStartE2EDuration="683.704361ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:00:00.68245874 +0000 UTC m=+2193.178127455" watchObservedRunningTime="2026-07-23 01:00:00.683704361 +0000 UTC m=+2193.179373074" Jul 23 01:00:01.675323 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:01.675291 2564 generic.go:356] "Generic (PLEG): container finished" podID="e020ab3f-88e3-4348-8368-6fad82386889" containerID="c238448d2ddc782ad88739a1b901451b972cbf0920668c122421bdb808d70709" exitCode=0 Jul 23 01:00:01.675937 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:01.675376 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-489ccc2d-29746140-clv4p" event={"ID":"e020ab3f-88e3-4348-8368-6fad82386889","Type":"ContainerDied","Data":"c238448d2ddc782ad88739a1b901451b972cbf0920668c122421bdb808d70709"} Jul 23 01:00:01.676940 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:01.676921 2564 generic.go:356] "Generic (PLEG): container finished" podID="e5d3dadb-1ef5-4c10-915c-38152ce5d26c" containerID="3f06b4db2757a91cc4d16929309a8719d28ecbf724b1e3fd16d47240b39cc9e4" exitCode=0 Jul 23 01:00:01.677001 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:01.676949 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-d7dq5" event={"ID":"e5d3dadb-1ef5-4c10-915c-38152ce5d26c","Type":"ContainerDied","Data":"3f06b4db2757a91cc4d16929309a8719d28ecbf724b1e3fd16d47240b39cc9e4"} Jul 23 01:00:02.661654 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:02.661621 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-489ccc2d-29746140-clv4p"] Jul 23 01:00:02.760460 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:02.760394 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:02.829609 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:02.829586 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-489ccc2d-29746140-clv4p" Jul 23 01:00:02.832652 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:02.832634 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-d7dq5" Jul 23 01:00:03.689390 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:03.689360 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-489ccc2d-29746140-clv4p" Jul 23 01:00:03.689590 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:03.689394 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-489ccc2d-29746140-clv4p" event={"ID":"e020ab3f-88e3-4348-8368-6fad82386889","Type":"ContainerDied","Data":"1582d584b24ece96c2a685ddd819fa7b15a926efbca55ada2a88e8637d8b9325"} Jul 23 01:00:03.689590 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:03.689443 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1582d584b24ece96c2a685ddd819fa7b15a926efbca55ada2a88e8637d8b9325" Jul 23 01:00:03.690822 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:03.690798 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-d7dq5" Jul 23 01:00:03.690957 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:03.690842 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29746140-d7dq5" event={"ID":"e5d3dadb-1ef5-4c10-915c-38152ce5d26c","Type":"ContainerDied","Data":"9d45d4b63179275cb34a92ab4d4d500d5e285e353d2070108603b7ab04ab2e9f"} Jul 23 01:00:03.690957 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:03.690886 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d45d4b63179275cb34a92ab4d4d500d5e285e353d2070108603b7ab04ab2e9f" Jul 23 01:00:03.724992 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:03.724968 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-489ccc2d-29746140-clv4p"] Jul 23 01:00:03.728478 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:03.728454 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-489ccc2d-29746140-clv4p"] Jul 23 01:00:04.073043 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:04.072973 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e020ab3f-88e3-4348-8368-6fad82386889" path="/var/lib/kubelet/pods/e020ab3f-88e3-4348-8368-6fad82386889/volumes" Jul 23 01:00:07.580393 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:07.580360 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:07.587251 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:07.587225 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr"] Jul 23 01:00:07.589610 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:07.589587 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-489ccc2d-sjjmr"] Jul 23 01:00:08.026851 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:08.026821 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:08.074623 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:08.074592 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bd4b182-ee4b-4208-afec-b0b3a63fbb77" path="/var/lib/kubelet/pods/0bd4b182-ee4b-4208-afec-b0b3a63fbb77/volumes" Jul 23 01:00:13.802297 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:13.802261 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:13.903394 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:13.903364 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:14.012502 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:14.012470 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:14.247721 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:14.247684 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:14.367667 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:14.367630 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:14.466699 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:14.466672 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:14.599583 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:14.599494 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:20.357858 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:20.357822 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:20.426501 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:20.426469 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:20.506574 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:20.506521 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:20.606474 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:20.606445 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:20.709905 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:20.709877 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:26.419396 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:26.419358 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:26.480110 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:26.480077 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:33.168941 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:33.168886 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:33.488497 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:33.488416 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:33.721208 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:33.721174 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:36.239758 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:36.239719 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:36.334505 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:36.334468 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:36.394260 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:36.394222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:36.598956 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:36.598862 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:36.939905 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:36.939874 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:37.041726 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:37.041694 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:40.756339 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:40.756287 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:00:52.343951 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:00:52.343870 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:01:19.192669 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.192637 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87"] Jul 23 01:01:19.193075 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.193031 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d3dadb-1ef5-4c10-915c-38152ce5d26c" containerName="cleanup" Jul 23 01:01:19.193075 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.193043 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d3dadb-1ef5-4c10-915c-38152ce5d26c" containerName="cleanup" Jul 23 01:01:19.193075 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.193053 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e020ab3f-88e3-4348-8368-6fad82386889" containerName="cleanup" Jul 23 01:01:19.193075 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.193059 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e020ab3f-88e3-4348-8368-6fad82386889" containerName="cleanup" Jul 23 01:01:19.193218 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.193129 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e020ab3f-88e3-4348-8368-6fad82386889" containerName="cleanup" Jul 23 01:01:19.193218 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.193135 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d3dadb-1ef5-4c10-915c-38152ce5d26c" containerName="cleanup" Jul 23 01:01:19.196121 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.196106 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87" Jul 23 01:01:19.200024 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.199992 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zd6c\"" Jul 23 01:01:19.200024 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.200018 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 01:01:19.205290 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.205267 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87"] Jul 23 01:01:19.309201 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.309172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6aedcd3-499f-4c61-87f3-1d3f2fb6a175-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0828012d-79v87\" (UID: \"e6aedcd3-499f-4c61-87f3-1d3f2fb6a175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87" Jul 23 01:01:19.409972 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.409942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6aedcd3-499f-4c61-87f3-1d3f2fb6a175-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0828012d-79v87\" (UID: \"e6aedcd3-499f-4c61-87f3-1d3f2fb6a175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87" Jul 23 01:01:19.410568 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.410523 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6aedcd3-499f-4c61-87f3-1d3f2fb6a175-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0828012d-79v87\" (UID: \"e6aedcd3-499f-4c61-87f3-1d3f2fb6a175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87" Jul 23 01:01:19.508687 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.508620 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87" Jul 23 01:01:19.638640 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:19.638610 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87"] Jul 23 01:01:19.640225 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:01:19.640181 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6aedcd3_499f_4c61_87f3_1d3f2fb6a175.slice/crio-60ee9a6159037063f36ef4ccf390ef2d31a3914e5dc16319b82dda2a817c8b47 WatchSource:0}: Error finding container 60ee9a6159037063f36ef4ccf390ef2d31a3914e5dc16319b82dda2a817c8b47: Status 404 returned error can't find the container with id 60ee9a6159037063f36ef4ccf390ef2d31a3914e5dc16319b82dda2a817c8b47 Jul 23 01:01:20.122275 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:20.122192 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87" event={"ID":"e6aedcd3-499f-4c61-87f3-1d3f2fb6a175","Type":"ContainerStarted","Data":"038785a79efb3855f7da0a404e2803add2f3cd7e736125f4ecc3bd3c08480cff"} Jul 23 01:01:20.122275 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:20.122241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87" event={"ID":"e6aedcd3-499f-4c61-87f3-1d3f2fb6a175","Type":"ContainerStarted","Data":"60ee9a6159037063f36ef4ccf390ef2d31a3914e5dc16319b82dda2a817c8b47"} Jul 23 01:01:20.138317 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:20.138274 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87" podStartSLOduration=1.13825931 podStartE2EDuration="1.13825931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:01:20.137205245 +0000 UTC m=+2272.632873958" watchObservedRunningTime="2026-07-23 01:01:20.13825931 +0000 UTC m=+2272.633928022" Jul 23 01:01:21.129638 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:21.129601 2564 generic.go:356] "Generic (PLEG): container finished" podID="e6aedcd3-499f-4c61-87f3-1d3f2fb6a175" containerID="038785a79efb3855f7da0a404e2803add2f3cd7e736125f4ecc3bd3c08480cff" exitCode=0 Jul 23 01:01:21.130051 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:21.129653 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87" event={"ID":"e6aedcd3-499f-4c61-87f3-1d3f2fb6a175","Type":"ContainerDied","Data":"038785a79efb3855f7da0a404e2803add2f3cd7e736125f4ecc3bd3c08480cff"} Jul 23 01:01:22.262868 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:22.262845 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87" Jul 23 01:01:22.336175 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:22.336147 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6aedcd3-499f-4c61-87f3-1d3f2fb6a175-maas-api-service-ca\") pod \"e6aedcd3-499f-4c61-87f3-1d3f2fb6a175\" (UID: \"e6aedcd3-499f-4c61-87f3-1d3f2fb6a175\") " Jul 23 01:01:22.336480 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:22.336458 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6aedcd3-499f-4c61-87f3-1d3f2fb6a175-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e6aedcd3-499f-4c61-87f3-1d3f2fb6a175" (UID: "e6aedcd3-499f-4c61-87f3-1d3f2fb6a175"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:01:22.437966 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:22.437895 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6aedcd3-499f-4c61-87f3-1d3f2fb6a175-maas-api-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:01:23.142640 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:23.142604 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87" event={"ID":"e6aedcd3-499f-4c61-87f3-1d3f2fb6a175","Type":"ContainerDied","Data":"60ee9a6159037063f36ef4ccf390ef2d31a3914e5dc16319b82dda2a817c8b47"} Jul 23 01:01:23.142814 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:23.142653 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60ee9a6159037063f36ef4ccf390ef2d31a3914e5dc16319b82dda2a817c8b47" Jul 23 01:01:23.142814 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:23.142618 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87" Jul 23 01:01:23.834475 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:23.834429 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:01:23.934540 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:23.934508 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:01:34.661323 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:34.661290 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:01:39.270605 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:39.270567 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:01:39.277055 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:39.277013 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87"] Jul 23 01:01:39.279653 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:39.279629 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0828012d-79v87"] Jul 23 01:01:40.072769 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:40.072739 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6aedcd3-499f-4c61-87f3-1d3f2fb6a175" path="/var/lib/kubelet/pods/e6aedcd3-499f-4c61-87f3-1d3f2fb6a175/volumes" Jul 23 01:01:41.443264 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:41.443230 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:01:46.549575 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:01:46.549541 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:09.656809 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:09.656770 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:23.290086 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:23.290052 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:23.460501 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:23.460467 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:27.833569 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:27.833523 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:27.933839 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:27.933776 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:28.260906 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:28.260875 2564 scope.go:117] "RemoveContainer" containerID="569c4111b50d1cec54872a857bc4e5652cc20be6293eba418a709963790bfa9b" Jul 23 01:02:28.269668 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:28.269644 2564 scope.go:117] "RemoveContainer" containerID="8bff889211ebcf0411418b1c2b97b7adeb56951188b7144fa7ff55b0b784dc1c" Jul 23 01:02:28.973718 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:28.973676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:29.073798 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:29.073766 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:29.257352 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:29.257274 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:29.487003 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:29.486970 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:30.541566 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:30.541499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:30.635099 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:30.635065 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:30.733521 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:30.733491 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:31.456459 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:31.456426 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:31.549929 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:31.549896 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:31.650262 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:31.650235 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:33.014806 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:33.014773 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:35.165031 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:35.164997 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:35.217871 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:35.217840 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:38.689394 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:38.689359 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:38.782657 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:38.782629 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:41.253246 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:41.253216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:45.140059 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:45.140026 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:45.243415 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:45.243384 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:45.342291 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:45.342261 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:55.838177 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:55.838142 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:02:56.175833 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:02:56.175786 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:17.156134 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:17.156099 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:23.549279 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:23.549247 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:23.649681 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:23.649640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:23.891185 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:23.891153 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:24.206917 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:24.206828 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:24.307067 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:24.307033 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:27.188883 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:27.188851 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:27.290050 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:27.290017 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:27.427280 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:27.427247 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:27.680156 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:27.680129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:28.008793 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:28.008703 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:28.108757 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:28.108717 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:28.277335 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:28.277186 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 01:03:28.284232 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:28.284210 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 01:03:31.649196 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:31.649157 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:31.932035 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:31.931946 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:32.262285 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:32.262211 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:32.362725 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:32.362693 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:35.236232 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.236192 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:35.330421 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.330387 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:35.337541 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.337498 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g"] Jul 23 01:03:35.338139 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.338118 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6aedcd3-499f-4c61-87f3-1d3f2fb6a175" containerName="revoke-keys" Jul 23 01:03:35.338185 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.338144 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6aedcd3-499f-4c61-87f3-1d3f2fb6a175" containerName="revoke-keys" Jul 23 01:03:35.338292 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.338280 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6aedcd3-499f-4c61-87f3-1d3f2fb6a175" containerName="revoke-keys" Jul 23 01:03:35.341797 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.341775 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:03:35.344686 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.344647 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-bb9f10-serving-cert\"" Jul 23 01:03:35.344816 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.344759 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 01:03:35.345944 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.345924 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sq2ww\"" Jul 23 01:03:35.346056 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.345945 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 01:03:35.354598 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.354575 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g"] Jul 23 01:03:35.396287 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.396257 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g\" (UID: \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:03:35.396428 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.396295 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkkgp\" (UniqueName: \"kubernetes.io/projected/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-kube-api-access-fkkgp\") pod \"maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g\" (UID: \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:03:35.396428 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.396343 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-maas-api-tls\") pod \"maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g\" (UID: \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:03:35.414898 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.414869 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:35.497114 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.497035 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g\" (UID: \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:03:35.497114 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.497079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkkgp\" (UniqueName: \"kubernetes.io/projected/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-kube-api-access-fkkgp\") pod \"maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g\" (UID: \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:03:35.497322 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.497132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-maas-api-tls\") pod \"maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g\" (UID: \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:03:35.497384 ip-10-0-139-1 kubenswrapper[2564]: E0723 01:03:35.497362 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-serving-cert: secret "maas-api-e2e-inf-b-bb9f10-serving-cert" not found Jul 23 01:03:35.497459 ip-10-0-139-1 kubenswrapper[2564]: E0723 01:03:35.497445 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-maas-api-tls podName:ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b nodeName:}" failed. No retries permitted until 2026-07-23 01:03:35.99742681 +0000 UTC m=+2408.493095509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-maas-api-tls") pod "maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" (UID: "ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b") : secret "maas-api-e2e-inf-b-bb9f10-serving-cert" not found Jul 23 01:03:35.497890 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.497864 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g\" (UID: \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:03:35.507584 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.507560 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkkgp\" (UniqueName: \"kubernetes.io/projected/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-kube-api-access-fkkgp\") pod \"maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g\" (UID: \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:03:35.999897 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:35.999867 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:36.002080 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:36.002059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-maas-api-tls\") pod \"maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g\" (UID: \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:03:36.004570 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:36.004515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-maas-api-tls\") pod \"maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g\" (UID: \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:03:36.099877 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:36.099848 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:36.199820 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:36.199785 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:36.255152 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:36.255078 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:03:36.392983 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:36.392958 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g"] Jul 23 01:03:36.394032 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:03:36.394006 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded1f0a72_0cf6_44cf_81ca_c1ff98065b8b.slice/crio-78f84d51d542c26efaece09a63aaee41dd7efc28c06d70ed075b353278cd7eac WatchSource:0}: Error finding container 78f84d51d542c26efaece09a63aaee41dd7efc28c06d70ed075b353278cd7eac: Status 404 returned error can't find the container with id 78f84d51d542c26efaece09a63aaee41dd7efc28c06d70ed075b353278cd7eac Jul 23 01:03:36.395473 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:36.395453 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 01:03:36.921179 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:36.921141 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" event={"ID":"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b","Type":"ContainerStarted","Data":"1deb5dd4f47bc5fd0ad678545019a7f962c5b4ded76212db78977ad3ea29eab8"} Jul 23 01:03:36.921324 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:36.921207 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:03:36.921324 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:36.921225 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" event={"ID":"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b","Type":"ContainerStarted","Data":"78f84d51d542c26efaece09a63aaee41dd7efc28c06d70ed075b353278cd7eac"} Jul 23 01:03:36.940503 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:36.940459 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" podStartSLOduration=1.540417065 podStartE2EDuration="1.940446588s" totalImagesPullingTime="400.029523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:03:36.939504302 +0000 UTC m=+2409.435173016" watchObservedRunningTime="2026-07-23 01:03:36.940446588 +0000 UTC m=+2409.436115300" Jul 23 01:03:39.628758 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:39.628703 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:42.932329 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:42.932301 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:03:48.777087 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:48.777036 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:48.880976 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:48.880929 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:56.560341 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:56.560249 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:56.813752 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:56.813670 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:03:56.911340 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:03:56.911297 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:04:04.119413 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:04.119374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:04:26.081770 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:26.081729 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:04:34.078020 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.077988 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7"] Jul 23 01:04:34.082170 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.082148 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.086550 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.086507 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bb9f10\"/\"test-model-bb9f10-kserve-self-signed-certs\"" Jul 23 01:04:34.086871 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.086515 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bb9f10\"/\"openshift-service-ca.crt\"" Jul 23 01:04:34.086979 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.086579 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bb9f10\"/\"kube-root-ca.crt\"" Jul 23 01:04:34.087235 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.087215 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bb9f10\"/\"default-dockercfg-vhxfc\"" Jul 23 01:04:34.090924 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.090905 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7"] Jul 23 01:04:34.120291 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.120263 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:04:34.204650 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.204626 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-tmp-dir\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.204752 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.204666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-home\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.204752 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.204695 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-dshm\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.204828 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.204767 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930c8423-083d-4a64-903a-14bd065a97b2-tls-certs\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.204828 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.204802 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-model-cache\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.204895 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.204861 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9vpn\" (UniqueName: \"kubernetes.io/projected/930c8423-083d-4a64-903a-14bd065a97b2-kube-api-access-r9vpn\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.305325 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.305298 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9vpn\" (UniqueName: \"kubernetes.io/projected/930c8423-083d-4a64-903a-14bd065a97b2-kube-api-access-r9vpn\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.305437 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.305339 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-tmp-dir\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.305437 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.305374 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-home\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.305437 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.305405 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-dshm\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.305631 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.305437 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930c8423-083d-4a64-903a-14bd065a97b2-tls-certs\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.305631 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.305458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-model-cache\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.305822 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.305796 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-tmp-dir\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.305894 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.305830 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-home\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.305894 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.305876 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-model-cache\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.307730 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.307708 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-dshm\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.307987 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.307969 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930c8423-083d-4a64-903a-14bd065a97b2-tls-certs\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.313606 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.313589 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9vpn\" (UniqueName: \"kubernetes.io/projected/930c8423-083d-4a64-903a-14bd065a97b2-kube-api-access-r9vpn\") pod \"test-model-bb9f10-kserve-c48b9cf8f-bbwx7\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.392861 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.392829 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:34.525972 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:34.525937 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7"] Jul 23 01:04:34.526760 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:04:34.526732 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod930c8423_083d_4a64_903a_14bd065a97b2.slice/crio-6534aa6999618a3e317afe8aada3e68727dbb22328d95ae8a2a84b51ffc909be WatchSource:0}: Error finding container 6534aa6999618a3e317afe8aada3e68727dbb22328d95ae8a2a84b51ffc909be: Status 404 returned error can't find the container with id 6534aa6999618a3e317afe8aada3e68727dbb22328d95ae8a2a84b51ffc909be Jul 23 01:04:35.255482 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:35.255443 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" event={"ID":"930c8423-083d-4a64-903a-14bd065a97b2","Type":"ContainerStarted","Data":"80fc342d46d2ab188bd4bb10968129b4e7c86ab1a0c62beca13f07306a6b46bd"} Jul 23 01:04:35.255894 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:35.255493 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" event={"ID":"930c8423-083d-4a64-903a-14bd065a97b2","Type":"ContainerStarted","Data":"6534aa6999618a3e317afe8aada3e68727dbb22328d95ae8a2a84b51ffc909be"} Jul 23 01:04:35.274850 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:35.274800 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" podStartSLOduration=1.2747845469999999 podStartE2EDuration="1.274784547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:04:35.271957 +0000 UTC m=+2467.767625717" watchObservedRunningTime="2026-07-23 01:04:35.274784547 +0000 UTC m=+2467.770453262" Jul 23 01:04:44.393856 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:44.393821 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:44.394226 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:44.393894 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:44.406663 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:44.406640 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:04:45.335787 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:04:45.335755 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:05:14.839812 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:14.839765 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:15.698239 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:15.698207 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7"] Jul 23 01:05:15.711751 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:15.711714 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:15.845566 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:15.845508 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q"] Jul 23 01:05:15.848791 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:15.848767 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q" Jul 23 01:05:15.851391 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:15.851371 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zd6c\"" Jul 23 01:05:15.856586 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:15.856521 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q"] Jul 23 01:05:16.013142 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:16.013049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q\" (UID: \"11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q" Jul 23 01:05:16.114692 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:16.114655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q\" (UID: \"11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q" Jul 23 01:05:16.115300 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:16.115281 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q\" (UID: \"11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q" Jul 23 01:05:16.164067 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:16.164030 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q" Jul 23 01:05:16.306974 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:16.306945 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q"] Jul 23 01:05:16.307416 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:05:16.307395 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f2e622_27e3_4bd8_bfb1_7ab58db0a7f4.slice/crio-969e77747fb9c2eec0377ede5609ec7d91c996d581b5d1cf9ca8e774434f1d24 WatchSource:0}: Error finding container 969e77747fb9c2eec0377ede5609ec7d91c996d581b5d1cf9ca8e774434f1d24: Status 404 returned error can't find the container with id 969e77747fb9c2eec0377ede5609ec7d91c996d581b5d1cf9ca8e774434f1d24 Jul 23 01:05:16.366549 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:16.366490 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:16.467191 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:16.467136 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:16.511481 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:16.511388 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q" event={"ID":"11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4","Type":"ContainerStarted","Data":"969e77747fb9c2eec0377ede5609ec7d91c996d581b5d1cf9ca8e774434f1d24"} Jul 23 01:05:17.519300 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:17.519266 2564 generic.go:356] "Generic (PLEG): container finished" podID="11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4" containerID="4fd9123250b29b03706a10db3c9ab28b2e99b3d794eaf938105e65da19ca6f00" exitCode=0 Jul 23 01:05:17.519761 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:17.519351 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q" event={"ID":"11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4","Type":"ContainerDied","Data":"4fd9123250b29b03706a10db3c9ab28b2e99b3d794eaf938105e65da19ca6f00"} Jul 23 01:05:18.670231 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:18.670196 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q" Jul 23 01:05:18.844644 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:18.844518 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4-maas-api-service-ca\") pod \"11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4\" (UID: \"11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4\") " Jul 23 01:05:18.845101 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:18.845077 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4" (UID: "11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:05:18.946243 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:18.946206 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4-maas-api-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:05:19.533015 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:19.532931 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q" Jul 23 01:05:19.533015 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:19.532957 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q" event={"ID":"11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4","Type":"ContainerDied","Data":"969e77747fb9c2eec0377ede5609ec7d91c996d581b5d1cf9ca8e774434f1d24"} Jul 23 01:05:19.533015 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:19.533005 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="969e77747fb9c2eec0377ede5609ec7d91c996d581b5d1cf9ca8e774434f1d24" Jul 23 01:05:26.301498 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:26.301459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:30.720060 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:30.720015 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" podUID="930c8423-083d-4a64-903a-14bd065a97b2" containerName="main" containerID="cri-o://80fc342d46d2ab188bd4bb10968129b4e7c86ab1a0c62beca13f07306a6b46bd" gracePeriod=45 Jul 23 01:05:30.922158 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:30.922124 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:30.940154 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:30.940118 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q"] Jul 23 01:05:30.944441 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:30.944408 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ce8306-9ls5q"] Jul 23 01:05:30.980179 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:30.980097 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:05:31.065044 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.065006 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-model-cache\") pod \"930c8423-083d-4a64-903a-14bd065a97b2\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " Jul 23 01:05:31.065240 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.065095 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9vpn\" (UniqueName: \"kubernetes.io/projected/930c8423-083d-4a64-903a-14bd065a97b2-kube-api-access-r9vpn\") pod \"930c8423-083d-4a64-903a-14bd065a97b2\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " Jul 23 01:05:31.065240 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.065125 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-home\") pod \"930c8423-083d-4a64-903a-14bd065a97b2\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " Jul 23 01:05:31.065240 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.065157 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-tmp-dir\") pod \"930c8423-083d-4a64-903a-14bd065a97b2\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " Jul 23 01:05:31.065240 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.065195 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-dshm\") pod \"930c8423-083d-4a64-903a-14bd065a97b2\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " Jul 23 01:05:31.065463 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.065290 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930c8423-083d-4a64-903a-14bd065a97b2-tls-certs\") pod \"930c8423-083d-4a64-903a-14bd065a97b2\" (UID: \"930c8423-083d-4a64-903a-14bd065a97b2\") " Jul 23 01:05:31.065463 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.065330 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "930c8423-083d-4a64-903a-14bd065a97b2" (UID: "930c8423-083d-4a64-903a-14bd065a97b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:05:31.065463 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.065442 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-home" (OuterVolumeSpecName: "home") pod "930c8423-083d-4a64-903a-14bd065a97b2" (UID: "930c8423-083d-4a64-903a-14bd065a97b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:05:31.065463 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.065453 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "930c8423-083d-4a64-903a-14bd065a97b2" (UID: "930c8423-083d-4a64-903a-14bd065a97b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:05:31.065729 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.065710 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-home\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:05:31.065790 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.065734 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-tmp-dir\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:05:31.065790 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.065748 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-model-cache\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:05:31.067571 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.067518 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930c8423-083d-4a64-903a-14bd065a97b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "930c8423-083d-4a64-903a-14bd065a97b2" (UID: "930c8423-083d-4a64-903a-14bd065a97b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:05:31.067571 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.067522 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-dshm" (OuterVolumeSpecName: "dshm") pod "930c8423-083d-4a64-903a-14bd065a97b2" (UID: "930c8423-083d-4a64-903a-14bd065a97b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:05:31.067732 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.067598 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930c8423-083d-4a64-903a-14bd065a97b2-kube-api-access-r9vpn" (OuterVolumeSpecName: "kube-api-access-r9vpn") pod "930c8423-083d-4a64-903a-14bd065a97b2" (UID: "930c8423-083d-4a64-903a-14bd065a97b2"). InnerVolumeSpecName "kube-api-access-r9vpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:05:31.166999 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.166954 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9vpn\" (UniqueName: \"kubernetes.io/projected/930c8423-083d-4a64-903a-14bd065a97b2-kube-api-access-r9vpn\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:05:31.166999 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.166994 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930c8423-083d-4a64-903a-14bd065a97b2-dshm\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:05:31.166999 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.167011 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930c8423-083d-4a64-903a-14bd065a97b2-tls-certs\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:05:31.350307 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.350209 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:31.606117 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.606018 2564 generic.go:356] "Generic (PLEG): container finished" podID="930c8423-083d-4a64-903a-14bd065a97b2" containerID="80fc342d46d2ab188bd4bb10968129b4e7c86ab1a0c62beca13f07306a6b46bd" exitCode=0 Jul 23 01:05:31.606117 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.606075 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" Jul 23 01:05:31.606117 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.606103 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" event={"ID":"930c8423-083d-4a64-903a-14bd065a97b2","Type":"ContainerDied","Data":"80fc342d46d2ab188bd4bb10968129b4e7c86ab1a0c62beca13f07306a6b46bd"} Jul 23 01:05:31.606381 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.606156 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7" event={"ID":"930c8423-083d-4a64-903a-14bd065a97b2","Type":"ContainerDied","Data":"6534aa6999618a3e317afe8aada3e68727dbb22328d95ae8a2a84b51ffc909be"} Jul 23 01:05:31.606381 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.606178 2564 scope.go:117] "RemoveContainer" containerID="80fc342d46d2ab188bd4bb10968129b4e7c86ab1a0c62beca13f07306a6b46bd" Jul 23 01:05:31.616956 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.616935 2564 scope.go:117] "RemoveContainer" containerID="80fc342d46d2ab188bd4bb10968129b4e7c86ab1a0c62beca13f07306a6b46bd" Jul 23 01:05:31.617287 ip-10-0-139-1 kubenswrapper[2564]: E0723 01:05:31.617264 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80fc342d46d2ab188bd4bb10968129b4e7c86ab1a0c62beca13f07306a6b46bd\": container with ID starting with 80fc342d46d2ab188bd4bb10968129b4e7c86ab1a0c62beca13f07306a6b46bd not found: ID does not exist" containerID="80fc342d46d2ab188bd4bb10968129b4e7c86ab1a0c62beca13f07306a6b46bd" Jul 23 01:05:31.617334 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.617299 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80fc342d46d2ab188bd4bb10968129b4e7c86ab1a0c62beca13f07306a6b46bd"} err="failed to get container status \"80fc342d46d2ab188bd4bb10968129b4e7c86ab1a0c62beca13f07306a6b46bd\": rpc error: code = NotFound desc = could not find container \"80fc342d46d2ab188bd4bb10968129b4e7c86ab1a0c62beca13f07306a6b46bd\": container with ID starting with 80fc342d46d2ab188bd4bb10968129b4e7c86ab1a0c62beca13f07306a6b46bd not found: ID does not exist" Jul 23 01:05:31.631017 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.630980 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7"] Jul 23 01:05:31.633458 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:31.633430 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-bb9f10/test-model-bb9f10-kserve-c48b9cf8f-bbwx7"] Jul 23 01:05:32.073155 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:32.073103 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4" path="/var/lib/kubelet/pods/11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4/volumes" Jul 23 01:05:32.073799 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:32.073766 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="930c8423-083d-4a64-903a-14bd065a97b2" path="/var/lib/kubelet/pods/930c8423-083d-4a64-903a-14bd065a97b2/volumes" Jul 23 01:05:34.005137 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:34.005100 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:34.059206 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:34.059174 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:37.283788 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:37.283742 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:37.349415 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:37.349381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:37.548277 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:37.548191 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:37.779086 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:37.779052 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:37.939155 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:37.939119 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr"] Jul 23 01:05:37.939630 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:37.939614 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="930c8423-083d-4a64-903a-14bd065a97b2" containerName="main" Jul 23 01:05:37.939630 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:37.939630 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="930c8423-083d-4a64-903a-14bd065a97b2" containerName="main" Jul 23 01:05:37.939783 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:37.939640 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4" containerName="revoke-keys" Jul 23 01:05:37.939783 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:37.939646 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4" containerName="revoke-keys" Jul 23 01:05:37.939783 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:37.939712 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f2e622-27e3-4bd8-bfb1-7ab58db0a7f4" containerName="revoke-keys" Jul 23 01:05:37.939783 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:37.939723 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="930c8423-083d-4a64-903a-14bd065a97b2" containerName="main" Jul 23 01:05:37.942696 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:37.942679 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr" Jul 23 01:05:37.946604 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:37.946576 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zd6c\"" Jul 23 01:05:37.959053 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:37.959024 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr"] Jul 23 01:05:38.035960 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:38.035918 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d86875-07cd-4bc9-986c-f4636730cdf2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr\" (UID: \"41d86875-07cd-4bc9-986c-f4636730cdf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr" Jul 23 01:05:38.137355 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:38.137313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d86875-07cd-4bc9-986c-f4636730cdf2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr\" (UID: \"41d86875-07cd-4bc9-986c-f4636730cdf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr" Jul 23 01:05:38.138003 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:38.137978 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d86875-07cd-4bc9-986c-f4636730cdf2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr\" (UID: \"41d86875-07cd-4bc9-986c-f4636730cdf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr" Jul 23 01:05:38.253745 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:38.253648 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr" Jul 23 01:05:38.400106 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:38.400073 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr"] Jul 23 01:05:38.401954 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:05:38.401924 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d86875_07cd_4bc9_986c_f4636730cdf2.slice/crio-8aefcbaa1f6cd32b3760bff43927ac9880b37e378cef84652ed7aff1ac4ad45c WatchSource:0}: Error finding container 8aefcbaa1f6cd32b3760bff43927ac9880b37e378cef84652ed7aff1ac4ad45c: Status 404 returned error can't find the container with id 8aefcbaa1f6cd32b3760bff43927ac9880b37e378cef84652ed7aff1ac4ad45c Jul 23 01:05:38.611276 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:38.611165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:38.650519 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:38.650482 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr" event={"ID":"41d86875-07cd-4bc9-986c-f4636730cdf2","Type":"ContainerStarted","Data":"8aefcbaa1f6cd32b3760bff43927ac9880b37e378cef84652ed7aff1ac4ad45c"} Jul 23 01:05:38.718054 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:38.718011 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:39.658006 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:39.657970 2564 generic.go:356] "Generic (PLEG): container finished" podID="41d86875-07cd-4bc9-986c-f4636730cdf2" containerID="108fc4c4c7ef2d0f4fb995bbb7d03ef2e39f4cedf99234a989f772d22cd82197" exitCode=0 Jul 23 01:05:39.658418 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:39.658061 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr" event={"ID":"41d86875-07cd-4bc9-986c-f4636730cdf2","Type":"ContainerDied","Data":"108fc4c4c7ef2d0f4fb995bbb7d03ef2e39f4cedf99234a989f772d22cd82197"} Jul 23 01:05:40.806310 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:40.806281 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr" Jul 23 01:05:40.965493 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:40.965379 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d86875-07cd-4bc9-986c-f4636730cdf2-maas-api-service-ca\") pod \"41d86875-07cd-4bc9-986c-f4636730cdf2\" (UID: \"41d86875-07cd-4bc9-986c-f4636730cdf2\") " Jul 23 01:05:40.965805 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:40.965772 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d86875-07cd-4bc9-986c-f4636730cdf2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41d86875-07cd-4bc9-986c-f4636730cdf2" (UID: "41d86875-07cd-4bc9-986c-f4636730cdf2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:05:41.066486 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:41.066444 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41d86875-07cd-4bc9-986c-f4636730cdf2-maas-api-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:05:41.671275 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:41.671242 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr" Jul 23 01:05:41.671457 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:41.671284 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr" event={"ID":"41d86875-07cd-4bc9-986c-f4636730cdf2","Type":"ContainerDied","Data":"8aefcbaa1f6cd32b3760bff43927ac9880b37e378cef84652ed7aff1ac4ad45c"} Jul 23 01:05:41.671457 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:41.671332 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aefcbaa1f6cd32b3760bff43927ac9880b37e378cef84652ed7aff1ac4ad45c" Jul 23 01:05:48.591787 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:48.591725 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g"] Jul 23 01:05:48.592307 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:48.591972 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" podUID="ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b" containerName="maas-api" containerID="cri-o://1deb5dd4f47bc5fd0ad678545019a7f962c5b4ded76212db78977ad3ea29eab8" gracePeriod=30 Jul 23 01:05:48.615574 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:48.615518 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:49.055167 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.055138 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:05:49.147917 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.147818 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-maas-api-tls\") pod \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\" (UID: \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\") " Jul 23 01:05:49.148117 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.147980 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkkgp\" (UniqueName: \"kubernetes.io/projected/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-kube-api-access-fkkgp\") pod \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\" (UID: \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\") " Jul 23 01:05:49.148194 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.148121 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-openshift-service-ca\") pod \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\" (UID: \"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b\") " Jul 23 01:05:49.148436 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.148412 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b" (UID: "ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:05:49.150010 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.149984 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-kube-api-access-fkkgp" (OuterVolumeSpecName: "kube-api-access-fkkgp") pod "ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b" (UID: "ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b"). InnerVolumeSpecName "kube-api-access-fkkgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:05:49.150127 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.150006 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b" (UID: "ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:05:49.250094 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.250007 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-maas-api-tls\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:05:49.250094 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.250047 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkkgp\" (UniqueName: \"kubernetes.io/projected/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-kube-api-access-fkkgp\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:05:49.250094 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.250063 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b-openshift-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:05:49.728811 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.728765 2564 generic.go:356] "Generic (PLEG): container finished" podID="ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b" containerID="1deb5dd4f47bc5fd0ad678545019a7f962c5b4ded76212db78977ad3ea29eab8" exitCode=0 Jul 23 01:05:49.729329 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.728840 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" Jul 23 01:05:49.729329 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.728857 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" event={"ID":"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b","Type":"ContainerDied","Data":"1deb5dd4f47bc5fd0ad678545019a7f962c5b4ded76212db78977ad3ea29eab8"} Jul 23 01:05:49.729329 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.728990 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g" event={"ID":"ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b","Type":"ContainerDied","Data":"78f84d51d542c26efaece09a63aaee41dd7efc28c06d70ed075b353278cd7eac"} Jul 23 01:05:49.729329 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.729024 2564 scope.go:117] "RemoveContainer" containerID="1deb5dd4f47bc5fd0ad678545019a7f962c5b4ded76212db78977ad3ea29eab8" Jul 23 01:05:49.739969 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.739945 2564 scope.go:117] "RemoveContainer" containerID="1deb5dd4f47bc5fd0ad678545019a7f962c5b4ded76212db78977ad3ea29eab8" Jul 23 01:05:49.740312 ip-10-0-139-1 kubenswrapper[2564]: E0723 01:05:49.740291 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1deb5dd4f47bc5fd0ad678545019a7f962c5b4ded76212db78977ad3ea29eab8\": container with ID starting with 1deb5dd4f47bc5fd0ad678545019a7f962c5b4ded76212db78977ad3ea29eab8 not found: ID does not exist" containerID="1deb5dd4f47bc5fd0ad678545019a7f962c5b4ded76212db78977ad3ea29eab8" Jul 23 01:05:49.740364 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.740323 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1deb5dd4f47bc5fd0ad678545019a7f962c5b4ded76212db78977ad3ea29eab8"} err="failed to get container status \"1deb5dd4f47bc5fd0ad678545019a7f962c5b4ded76212db78977ad3ea29eab8\": rpc error: code = NotFound desc = could not find container \"1deb5dd4f47bc5fd0ad678545019a7f962c5b4ded76212db78977ad3ea29eab8\": container with ID starting with 1deb5dd4f47bc5fd0ad678545019a7f962c5b4ded76212db78977ad3ea29eab8 not found: ID does not exist" Jul 23 01:05:49.753120 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.753079 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g"] Jul 23 01:05:49.756739 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:49.756703 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bb9f10-679cd5747d-hkl6g"] Jul 23 01:05:50.073883 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:50.073800 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b" path="/var/lib/kubelet/pods/ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b/volumes" Jul 23 01:05:53.048092 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:53.048053 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:53.051899 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:53.051867 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr"] Jul 23 01:05:53.054718 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:53.054695 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bb9f10-2jwmr"] Jul 23 01:05:53.433837 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:53.433803 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:05:54.072624 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:54.072584 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41d86875-07cd-4bc9-986c-f4636730cdf2" path="/var/lib/kubelet/pods/41d86875-07cd-4bc9-986c-f4636730cdf2/volumes" Jul 23 01:05:59.447105 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:05:59.447067 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:00.830907 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:00.830855 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:00.971012 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:00.970971 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:03.863582 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:03.863550 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:03.956743 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:03.956710 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:04.015620 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:04.015576 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:04.176927 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:04.176899 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:04.519609 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:04.519491 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:04.620652 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:04.620617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:08.383145 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:08.383109 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:08.844569 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:08.844509 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:09.147256 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:09.147223 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:09.406887 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:09.406785 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:11.931507 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:11.931474 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:12.033595 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:12.033514 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:12.133047 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:12.133020 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:12.466698 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:12.466663 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:12.594928 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:12.594900 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:12.694110 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:12.694057 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:16.415698 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:16.415666 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:24.434309 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.434278 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c"] Jul 23 01:06:24.434746 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.434695 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41d86875-07cd-4bc9-986c-f4636730cdf2" containerName="revoke-keys" Jul 23 01:06:24.434746 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.434708 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="41d86875-07cd-4bc9-986c-f4636730cdf2" containerName="revoke-keys" Jul 23 01:06:24.434746 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.434722 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b" containerName="maas-api" Jul 23 01:06:24.434746 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.434728 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b" containerName="maas-api" Jul 23 01:06:24.434872 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.434788 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="41d86875-07cd-4bc9-986c-f4636730cdf2" containerName="revoke-keys" Jul 23 01:06:24.434872 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.434796 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed1f0a72-0cf6-44cf-81ca-c1ff98065b8b" containerName="maas-api" Jul 23 01:06:24.437919 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.437905 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c" Jul 23 01:06:24.440755 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.440685 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zd6c\"" Jul 23 01:06:24.440755 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.440685 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 01:06:24.445887 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.445521 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c"] Jul 23 01:06:24.563962 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.563930 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc7526e-ff02-4f95-a21b-e1d4b3b8257f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c\" (UID: \"5bc7526e-ff02-4f95-a21b-e1d4b3b8257f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c" Jul 23 01:06:24.664700 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.664674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc7526e-ff02-4f95-a21b-e1d4b3b8257f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c\" (UID: \"5bc7526e-ff02-4f95-a21b-e1d4b3b8257f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c" Jul 23 01:06:24.665312 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.665287 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc7526e-ff02-4f95-a21b-e1d4b3b8257f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c\" (UID: \"5bc7526e-ff02-4f95-a21b-e1d4b3b8257f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c" Jul 23 01:06:24.750272 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.750201 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c" Jul 23 01:06:24.881438 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.881413 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c"] Jul 23 01:06:24.882942 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:06:24.882915 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc7526e_ff02_4f95_a21b_e1d4b3b8257f.slice/crio-779596fc66bdcdc2eca2d929bca8b6e11f3f0cb5bb3aeafd7fe3558c7c39dcbf WatchSource:0}: Error finding container 779596fc66bdcdc2eca2d929bca8b6e11f3f0cb5bb3aeafd7fe3558c7c39dcbf: Status 404 returned error can't find the container with id 779596fc66bdcdc2eca2d929bca8b6e11f3f0cb5bb3aeafd7fe3558c7c39dcbf Jul 23 01:06:24.940178 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:24.940150 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c" event={"ID":"5bc7526e-ff02-4f95-a21b-e1d4b3b8257f","Type":"ContainerStarted","Data":"779596fc66bdcdc2eca2d929bca8b6e11f3f0cb5bb3aeafd7fe3558c7c39dcbf"} Jul 23 01:06:25.095036 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:25.094962 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:25.195801 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:25.195768 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:25.947784 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:25.947752 2564 generic.go:356] "Generic (PLEG): container finished" podID="5bc7526e-ff02-4f95-a21b-e1d4b3b8257f" containerID="12f467f1d76137748c1965a0d506ebf5f86aa64109c6834a51428216fa02eab4" exitCode=0 Jul 23 01:06:25.948171 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:25.947841 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c" event={"ID":"5bc7526e-ff02-4f95-a21b-e1d4b3b8257f","Type":"ContainerDied","Data":"12f467f1d76137748c1965a0d506ebf5f86aa64109c6834a51428216fa02eab4"} Jul 23 01:06:27.087772 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:27.087745 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c" Jul 23 01:06:27.184863 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:27.184838 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc7526e-ff02-4f95-a21b-e1d4b3b8257f-maas-api-service-ca\") pod \"5bc7526e-ff02-4f95-a21b-e1d4b3b8257f\" (UID: \"5bc7526e-ff02-4f95-a21b-e1d4b3b8257f\") " Jul 23 01:06:27.185157 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:27.185137 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc7526e-ff02-4f95-a21b-e1d4b3b8257f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5bc7526e-ff02-4f95-a21b-e1d4b3b8257f" (UID: "5bc7526e-ff02-4f95-a21b-e1d4b3b8257f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:06:27.286059 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:27.285989 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc7526e-ff02-4f95-a21b-e1d4b3b8257f-maas-api-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:06:27.960950 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:27.960918 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c" Jul 23 01:06:27.961126 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:27.960952 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c" event={"ID":"5bc7526e-ff02-4f95-a21b-e1d4b3b8257f","Type":"ContainerDied","Data":"779596fc66bdcdc2eca2d929bca8b6e11f3f0cb5bb3aeafd7fe3558c7c39dcbf"} Jul 23 01:06:27.961126 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:27.960998 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="779596fc66bdcdc2eca2d929bca8b6e11f3f0cb5bb3aeafd7fe3558c7c39dcbf" Jul 23 01:06:28.320435 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:28.320355 2564 scope.go:117] "RemoveContainer" containerID="c238448d2ddc782ad88739a1b901451b972cbf0920668c122421bdb808d70709" Jul 23 01:06:28.331187 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:28.331163 2564 scope.go:117] "RemoveContainer" containerID="f9a86b5c7c2b775414ac657ea5450be104017c84b0fbb347ce7d59f9dcfeca77" Jul 23 01:06:33.445738 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:33.445699 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:33.541687 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:33.541657 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:35.026829 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:35.026792 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:35.127677 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:35.127643 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:38.316783 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:38.316748 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:38.377770 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:38.377737 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:40.291981 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:40.291948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:44.556038 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:44.556000 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:44.557629 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:44.557606 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c"] Jul 23 01:06:44.562128 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:44.562106 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c2ab89-6t94c"] Jul 23 01:06:44.928155 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:44.928129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:46.072597 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:46.072559 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bc7526e-ff02-4f95-a21b-e1d4b3b8257f" path="/var/lib/kubelet/pods/5bc7526e-ff02-4f95-a21b-e1d4b3b8257f/volumes" Jul 23 01:06:50.661820 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:50.661717 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:50.723411 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:50.723369 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:50.893116 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:50.893080 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:51.112537 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:51.112446 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:51.232631 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:51.232602 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5"] Jul 23 01:06:51.232997 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:51.232985 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bc7526e-ff02-4f95-a21b-e1d4b3b8257f" containerName="revoke-keys" Jul 23 01:06:51.233045 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:51.232998 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bc7526e-ff02-4f95-a21b-e1d4b3b8257f" containerName="revoke-keys" Jul 23 01:06:51.233093 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:51.233070 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bc7526e-ff02-4f95-a21b-e1d4b3b8257f" containerName="revoke-keys" Jul 23 01:06:51.236125 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:51.236110 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5" Jul 23 01:06:51.238770 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:51.238729 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 01:06:51.238888 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:51.238782 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zd6c\"" Jul 23 01:06:51.244127 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:51.244100 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5"] Jul 23 01:06:51.295364 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:51.295335 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732af2b9-cf43-4409-908f-83ecb7edbc31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f05be8-wndg5\" (UID: \"732af2b9-cf43-4409-908f-83ecb7edbc31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5" Jul 23 01:06:51.396277 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:51.396251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732af2b9-cf43-4409-908f-83ecb7edbc31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f05be8-wndg5\" (UID: \"732af2b9-cf43-4409-908f-83ecb7edbc31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5" Jul 23 01:06:51.396878 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:51.396856 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732af2b9-cf43-4409-908f-83ecb7edbc31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f05be8-wndg5\" (UID: \"732af2b9-cf43-4409-908f-83ecb7edbc31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5" Jul 23 01:06:51.547587 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:51.547523 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5" Jul 23 01:06:51.682047 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:51.682016 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5"] Jul 23 01:06:51.683876 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:06:51.683853 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732af2b9_cf43_4409_908f_83ecb7edbc31.slice/crio-bebde6418ea5d4233265be10268c4e9d06d8fee1ac177fd6c5e173f68b1bfaad WatchSource:0}: Error finding container bebde6418ea5d4233265be10268c4e9d06d8fee1ac177fd6c5e173f68b1bfaad: Status 404 returned error can't find the container with id bebde6418ea5d4233265be10268c4e9d06d8fee1ac177fd6c5e173f68b1bfaad Jul 23 01:06:51.901025 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:51.900996 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:52.001979 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:52.001956 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:06:52.097911 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:52.097874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5" event={"ID":"732af2b9-cf43-4409-908f-83ecb7edbc31","Type":"ContainerStarted","Data":"379f0a5a80aa603e903a2ce029ee48c742a98706183c64aae28a32785866903f"} Jul 23 01:06:52.097911 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:52.097920 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5" event={"ID":"732af2b9-cf43-4409-908f-83ecb7edbc31","Type":"ContainerStarted","Data":"bebde6418ea5d4233265be10268c4e9d06d8fee1ac177fd6c5e173f68b1bfaad"} Jul 23 01:06:52.119297 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:52.119248 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5" podStartSLOduration=1.119233799 podStartE2EDuration="1.119233799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:06:52.117807046 +0000 UTC m=+2604.613475746" watchObservedRunningTime="2026-07-23 01:06:52.119233799 +0000 UTC m=+2604.614902512" Jul 23 01:06:53.104792 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:53.104757 2564 generic.go:356] "Generic (PLEG): container finished" podID="732af2b9-cf43-4409-908f-83ecb7edbc31" containerID="379f0a5a80aa603e903a2ce029ee48c742a98706183c64aae28a32785866903f" exitCode=0 Jul 23 01:06:53.105192 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:53.104843 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5" event={"ID":"732af2b9-cf43-4409-908f-83ecb7edbc31","Type":"ContainerDied","Data":"379f0a5a80aa603e903a2ce029ee48c742a98706183c64aae28a32785866903f"} Jul 23 01:06:54.239236 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:54.239208 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5" Jul 23 01:06:54.325644 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:54.325615 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732af2b9-cf43-4409-908f-83ecb7edbc31-maas-api-service-ca\") pod \"732af2b9-cf43-4409-908f-83ecb7edbc31\" (UID: \"732af2b9-cf43-4409-908f-83ecb7edbc31\") " Jul 23 01:06:54.325965 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:54.325943 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732af2b9-cf43-4409-908f-83ecb7edbc31-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "732af2b9-cf43-4409-908f-83ecb7edbc31" (UID: "732af2b9-cf43-4409-908f-83ecb7edbc31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:06:54.426559 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:54.426513 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732af2b9-cf43-4409-908f-83ecb7edbc31-maas-api-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:06:55.117423 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:55.117395 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5" Jul 23 01:06:55.117652 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:55.117392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5" event={"ID":"732af2b9-cf43-4409-908f-83ecb7edbc31","Type":"ContainerDied","Data":"bebde6418ea5d4233265be10268c4e9d06d8fee1ac177fd6c5e173f68b1bfaad"} Jul 23 01:06:55.117652 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:06:55.117546 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bebde6418ea5d4233265be10268c4e9d06d8fee1ac177fd6c5e173f68b1bfaad" Jul 23 01:07:06.709287 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:06.709247 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:11.323827 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:11.323792 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:11.328147 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:11.328048 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5"] Jul 23 01:07:11.330968 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:11.330943 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f05be8-wndg5"] Jul 23 01:07:11.715459 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:11.715430 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:12.073415 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:12.073329 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="732af2b9-cf43-4409-908f-83ecb7edbc31" path="/var/lib/kubelet/pods/732af2b9-cf43-4409-908f-83ecb7edbc31/volumes" Jul 23 01:07:17.490197 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:17.490167 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:17.554492 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:17.554458 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:17.783722 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:17.783642 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:17.993591 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:17.993557 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:18.347002 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:18.346967 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:18.661360 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:18.661337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:18.992492 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:18.992400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:21.299025 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.298985 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:21.373844 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.373814 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:21.454874 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.454838 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:21.457302 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.457282 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc"] Jul 23 01:07:21.457708 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.457696 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732af2b9-cf43-4409-908f-83ecb7edbc31" containerName="revoke-keys" Jul 23 01:07:21.457753 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.457710 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="732af2b9-cf43-4409-908f-83ecb7edbc31" containerName="revoke-keys" Jul 23 01:07:21.457838 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.457767 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="732af2b9-cf43-4409-908f-83ecb7edbc31" containerName="revoke-keys" Jul 23 01:07:21.460894 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.460879 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" Jul 23 01:07:21.463669 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.463645 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-7ee5a5-serving-cert\"" Jul 23 01:07:21.463778 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.463672 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sq2ww\"" Jul 23 01:07:21.463778 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.463711 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 01:07:21.464735 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.464721 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 01:07:21.472668 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.472645 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc"] Jul 23 01:07:21.560562 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.560454 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ccc9b-9902-4c90-b9de-c06204a36322-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc\" (UID: \"6e9ccc9b-9902-4c90-b9de-c06204a36322\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" Jul 23 01:07:21.560562 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.560503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9ccc9b-9902-4c90-b9de-c06204a36322-maas-api-tls\") pod \"maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc\" (UID: \"6e9ccc9b-9902-4c90-b9de-c06204a36322\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" Jul 23 01:07:21.560745 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.560690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb2nz\" (UniqueName: \"kubernetes.io/projected/6e9ccc9b-9902-4c90-b9de-c06204a36322-kube-api-access-lb2nz\") pod \"maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc\" (UID: \"6e9ccc9b-9902-4c90-b9de-c06204a36322\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" Jul 23 01:07:21.662080 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.662050 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb2nz\" (UniqueName: \"kubernetes.io/projected/6e9ccc9b-9902-4c90-b9de-c06204a36322-kube-api-access-lb2nz\") pod \"maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc\" (UID: \"6e9ccc9b-9902-4c90-b9de-c06204a36322\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" Jul 23 01:07:21.662266 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.662093 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ccc9b-9902-4c90-b9de-c06204a36322-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc\" (UID: \"6e9ccc9b-9902-4c90-b9de-c06204a36322\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" Jul 23 01:07:21.662266 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.662114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9ccc9b-9902-4c90-b9de-c06204a36322-maas-api-tls\") pod \"maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc\" (UID: \"6e9ccc9b-9902-4c90-b9de-c06204a36322\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" Jul 23 01:07:21.662786 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.662760 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ccc9b-9902-4c90-b9de-c06204a36322-openshift-service-ca\") pod \"maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc\" (UID: \"6e9ccc9b-9902-4c90-b9de-c06204a36322\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" Jul 23 01:07:21.664503 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.664483 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9ccc9b-9902-4c90-b9de-c06204a36322-maas-api-tls\") pod \"maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc\" (UID: \"6e9ccc9b-9902-4c90-b9de-c06204a36322\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" Jul 23 01:07:21.672847 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.672822 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:21.673853 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.673833 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb2nz\" (UniqueName: \"kubernetes.io/projected/6e9ccc9b-9902-4c90-b9de-c06204a36322-kube-api-access-lb2nz\") pod \"maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc\" (UID: \"6e9ccc9b-9902-4c90-b9de-c06204a36322\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" Jul 23 01:07:21.773518 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.773489 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" Jul 23 01:07:21.914690 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.914662 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc"] Jul 23 01:07:21.915737 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:07:21.915711 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9ccc9b_9902_4c90_b9de_c06204a36322.slice/crio-55e3732f2ba8c04c51e050fb876d3520a5f095860b4fb3e036b3cff1c2bb6ebd WatchSource:0}: Error finding container 55e3732f2ba8c04c51e050fb876d3520a5f095860b4fb3e036b3cff1c2bb6ebd: Status 404 returned error can't find the container with id 55e3732f2ba8c04c51e050fb876d3520a5f095860b4fb3e036b3cff1c2bb6ebd Jul 23 01:07:21.990279 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:21.990253 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:22.093337 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:22.093253 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:22.274091 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:22.274065 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" event={"ID":"6e9ccc9b-9902-4c90-b9de-c06204a36322","Type":"ContainerStarted","Data":"55e3732f2ba8c04c51e050fb876d3520a5f095860b4fb3e036b3cff1c2bb6ebd"} Jul 23 01:07:23.282042 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:23.282005 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" event={"ID":"6e9ccc9b-9902-4c90-b9de-c06204a36322","Type":"ContainerStarted","Data":"ddde4535d5b481c26e6c4c17f816d6f64f8b20bd86917129c753d3da50ea57e2"} Jul 23 01:07:23.282414 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:23.282062 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" Jul 23 01:07:23.303468 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:23.303422 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" podStartSLOduration=1.949110062 podStartE2EDuration="2.303406332s" totalImagesPullingTime="354.29627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:07:23.3018791 +0000 UTC m=+2635.797547833" watchObservedRunningTime="2026-07-23 01:07:23.303406332 +0000 UTC m=+2635.799075046" Jul 23 01:07:25.826938 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:25.826901 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:26.157027 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:26.156993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:26.484137 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:26.484039 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:26.595642 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:26.595601 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:28.364401 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:28.364368 2564 scope.go:117] "RemoveContainer" containerID="038785a79efb3855f7da0a404e2803add2f3cd7e736125f4ecc3bd3c08480cff" Jul 23 01:07:28.649508 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:28.649473 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:28.749159 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:28.749123 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:29.293301 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:29.293275 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" Jul 23 01:07:29.371014 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:29.370977 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:29.455061 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:29.455022 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:29.516918 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:29.516888 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:29.616581 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:29.616482 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:30.155402 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:30.155369 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:30.255945 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:30.255897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:33.869082 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:33.869053 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:42.695747 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:42.695712 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:43.064297 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:43.064212 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:43.166402 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:43.166370 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:50.969572 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:50.969492 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:07:51.069992 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:07:51.069962 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:08:12.642820 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:08:12.642779 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:08:28.323587 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:08:28.323460 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 01:08:28.332545 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:08:28.332500 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 01:09:02.097420 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:09:02.097390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:10:16.192825 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:16.192730 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:10:27.795187 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:27.795151 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:10:28.786671 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:28.786641 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:10:36.899803 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:36.899768 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6"] Jul 23 01:10:36.903379 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:36.903359 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:36.907807 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:36.907785 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7ee5a5\"/\"kube-root-ca.crt\"" Jul 23 01:10:36.907918 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:36.907785 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7ee5a5\"/\"sub-test-model-7ee5a5-kserve-self-signed-certs\"" Jul 23 01:10:36.907918 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:36.907785 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7ee5a5\"/\"default-dockercfg-t96hh\"" Jul 23 01:10:36.907918 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:36.907785 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7ee5a5\"/\"openshift-service-ca.crt\"" Jul 23 01:10:36.913312 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:36.913287 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6"] Jul 23 01:10:36.932102 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:36.932070 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:10:37.033555 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.033504 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-home\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.033555 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.033549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9724c2a8-62d7-4d4a-8449-953855e11571-tls-certs\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.033731 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.033624 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-model-cache\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.033731 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.033654 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-tmp-dir\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.033731 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.033724 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-dshm\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.033891 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.033823 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97bf\" (UniqueName: \"kubernetes.io/projected/9724c2a8-62d7-4d4a-8449-953855e11571-kube-api-access-p97bf\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.134542 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.134494 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-home\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.134661 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.134549 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9724c2a8-62d7-4d4a-8449-953855e11571-tls-certs\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.134661 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.134580 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-model-cache\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.134661 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.134595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-tmp-dir\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.134661 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.134635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-dshm\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.134835 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.134709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p97bf\" (UniqueName: \"kubernetes.io/projected/9724c2a8-62d7-4d4a-8449-953855e11571-kube-api-access-p97bf\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.135023 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.134987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-model-cache\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.135122 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.135024 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-home\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.135122 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.135072 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-tmp-dir\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.136778 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.136760 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-dshm\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.136967 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.136949 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9724c2a8-62d7-4d4a-8449-953855e11571-tls-certs\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.143602 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.143582 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97bf\" (UniqueName: \"kubernetes.io/projected/9724c2a8-62d7-4d4a-8449-953855e11571-kube-api-access-p97bf\") pod \"sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.213819 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.213750 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:37.342223 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.342192 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6"] Jul 23 01:10:37.343355 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:10:37.343332 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9724c2a8_62d7_4d4a_8449_953855e11571.slice/crio-d4268be5f4a625bc0aa9ef91808ad0965738643631909d5810d226e62427dfd9 WatchSource:0}: Error finding container d4268be5f4a625bc0aa9ef91808ad0965738643631909d5810d226e62427dfd9: Status 404 returned error can't find the container with id d4268be5f4a625bc0aa9ef91808ad0965738643631909d5810d226e62427dfd9 Jul 23 01:10:37.379261 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:37.379230 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" event={"ID":"9724c2a8-62d7-4d4a-8449-953855e11571","Type":"ContainerStarted","Data":"d4268be5f4a625bc0aa9ef91808ad0965738643631909d5810d226e62427dfd9"} Jul 23 01:10:38.386312 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:38.386276 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" event={"ID":"9724c2a8-62d7-4d4a-8449-953855e11571","Type":"ContainerStarted","Data":"1095c9f7aa0ec76dd9bdacffb8494d381aa9d679454ade03c55e8c27bdceb2b2"} Jul 23 01:10:38.405202 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:38.405156 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" podStartSLOduration=2.4051425269999998 podStartE2EDuration="2.405142527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:10:38.402651831 +0000 UTC m=+2830.898320545" watchObservedRunningTime="2026-07-23 01:10:38.405142527 +0000 UTC m=+2830.900811238" Jul 23 01:10:40.343004 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:40.342973 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:10:47.214836 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:47.214798 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:47.215323 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:47.214859 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:47.227670 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:47.227644 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:47.455855 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:47.455829 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:10:56.916900 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:10:56.916862 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:11:28.413306 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:28.413272 2564 scope.go:117] "RemoveContainer" containerID="4fd9123250b29b03706a10db3c9ab28b2e99b3d794eaf938105e65da19ca6f00" Jul 23 01:11:43.673281 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:43.673249 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6"] Jul 23 01:11:43.694311 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:43.694282 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:11:45.216965 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:45.216934 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:11:51.414263 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:51.414227 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:11:58.693634 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:58.693573 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" podUID="9724c2a8-62d7-4d4a-8449-953855e11571" containerName="main" containerID="cri-o://1095c9f7aa0ec76dd9bdacffb8494d381aa9d679454ade03c55e8c27bdceb2b2" gracePeriod=45 Jul 23 01:11:58.860633 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:58.860589 2564 generic.go:356] "Generic (PLEG): container finished" podID="9724c2a8-62d7-4d4a-8449-953855e11571" containerID="1095c9f7aa0ec76dd9bdacffb8494d381aa9d679454ade03c55e8c27bdceb2b2" exitCode=0 Jul 23 01:11:58.860892 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:58.860793 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" event={"ID":"9724c2a8-62d7-4d4a-8449-953855e11571","Type":"ContainerDied","Data":"1095c9f7aa0ec76dd9bdacffb8494d381aa9d679454ade03c55e8c27bdceb2b2"} Jul 23 01:11:58.957992 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:58.957934 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:11:59.094442 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.094414 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-dshm\") pod \"9724c2a8-62d7-4d4a-8449-953855e11571\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " Jul 23 01:11:59.094629 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.094485 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-home\") pod \"9724c2a8-62d7-4d4a-8449-953855e11571\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " Jul 23 01:11:59.094629 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.094516 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-tmp-dir\") pod \"9724c2a8-62d7-4d4a-8449-953855e11571\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " Jul 23 01:11:59.094629 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.094597 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p97bf\" (UniqueName: \"kubernetes.io/projected/9724c2a8-62d7-4d4a-8449-953855e11571-kube-api-access-p97bf\") pod \"9724c2a8-62d7-4d4a-8449-953855e11571\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " Jul 23 01:11:59.094810 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.094648 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9724c2a8-62d7-4d4a-8449-953855e11571-tls-certs\") pod \"9724c2a8-62d7-4d4a-8449-953855e11571\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " Jul 23 01:11:59.094810 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.094674 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-model-cache\") pod \"9724c2a8-62d7-4d4a-8449-953855e11571\" (UID: \"9724c2a8-62d7-4d4a-8449-953855e11571\") " Jul 23 01:11:59.094810 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.094715 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-home" (OuterVolumeSpecName: "home") pod "9724c2a8-62d7-4d4a-8449-953855e11571" (UID: "9724c2a8-62d7-4d4a-8449-953855e11571"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:11:59.094810 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.094753 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9724c2a8-62d7-4d4a-8449-953855e11571" (UID: "9724c2a8-62d7-4d4a-8449-953855e11571"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:11:59.095027 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.094980 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-home\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:11:59.095027 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.094990 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-model-cache" (OuterVolumeSpecName: "model-cache") pod "9724c2a8-62d7-4d4a-8449-953855e11571" (UID: "9724c2a8-62d7-4d4a-8449-953855e11571"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:11:59.095027 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.095025 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-tmp-dir\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:11:59.096857 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.096825 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-dshm" (OuterVolumeSpecName: "dshm") pod "9724c2a8-62d7-4d4a-8449-953855e11571" (UID: "9724c2a8-62d7-4d4a-8449-953855e11571"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:11:59.096857 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.096827 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9724c2a8-62d7-4d4a-8449-953855e11571-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9724c2a8-62d7-4d4a-8449-953855e11571" (UID: "9724c2a8-62d7-4d4a-8449-953855e11571"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:11:59.096995 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.096882 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9724c2a8-62d7-4d4a-8449-953855e11571-kube-api-access-p97bf" (OuterVolumeSpecName: "kube-api-access-p97bf") pod "9724c2a8-62d7-4d4a-8449-953855e11571" (UID: "9724c2a8-62d7-4d4a-8449-953855e11571"). InnerVolumeSpecName "kube-api-access-p97bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:11:59.195752 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.195725 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p97bf\" (UniqueName: \"kubernetes.io/projected/9724c2a8-62d7-4d4a-8449-953855e11571-kube-api-access-p97bf\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:11:59.195752 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.195749 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9724c2a8-62d7-4d4a-8449-953855e11571-tls-certs\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:11:59.195888 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.195758 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-model-cache\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:11:59.195888 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.195766 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9724c2a8-62d7-4d4a-8449-953855e11571-dshm\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:11:59.868343 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.868311 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" Jul 23 01:11:59.868818 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.868358 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6" event={"ID":"9724c2a8-62d7-4d4a-8449-953855e11571","Type":"ContainerDied","Data":"d4268be5f4a625bc0aa9ef91808ad0965738643631909d5810d226e62427dfd9"} Jul 23 01:11:59.868818 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.868408 2564 scope.go:117] "RemoveContainer" containerID="1095c9f7aa0ec76dd9bdacffb8494d381aa9d679454ade03c55e8c27bdceb2b2" Jul 23 01:11:59.895921 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.895849 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6"] Jul 23 01:11:59.899422 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:11:59.899398 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7ee5a5/sub-test-model-7ee5a5-kserve-5dcb98ccc9-k4hw6"] Jul 23 01:12:00.073284 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:00.073256 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9724c2a8-62d7-4d4a-8449-953855e11571" path="/var/lib/kubelet/pods/9724c2a8-62d7-4d4a-8449-953855e11571/volumes" Jul 23 01:12:09.090190 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.090156 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd"] Jul 23 01:12:09.090687 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.090669 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9724c2a8-62d7-4d4a-8449-953855e11571" containerName="main" Jul 23 01:12:09.090687 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.090690 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9724c2a8-62d7-4d4a-8449-953855e11571" containerName="main" Jul 23 01:12:09.090854 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.090777 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9724c2a8-62d7-4d4a-8449-953855e11571" containerName="main" Jul 23 01:12:09.093953 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.093935 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.098447 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.098424 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d6869a\"/\"rate-test-model-d6869a-kserve-self-signed-certs\"" Jul 23 01:12:09.098447 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.098444 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d6869a\"/\"openshift-service-ca.crt\"" Jul 23 01:12:09.098653 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.098431 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d6869a\"/\"default-dockercfg-d58v2\"" Jul 23 01:12:09.098653 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.098488 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d6869a\"/\"kube-root-ca.crt\"" Jul 23 01:12:09.103349 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.103325 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd"] Jul 23 01:12:09.114438 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.114413 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:12:09.176470 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.176447 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9cd3edd-e771-4821-a270-47c6d8bf6568-tls-certs\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.176677 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.176653 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-model-cache\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.176763 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.176750 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-dshm\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.176830 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.176802 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-tmp-dir\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.176985 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.176961 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8fr2\" (UniqueName: \"kubernetes.io/projected/f9cd3edd-e771-4821-a270-47c6d8bf6568-kube-api-access-k8fr2\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.177057 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.177034 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-home\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.277686 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.277660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-model-cache\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.277811 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.277702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-dshm\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.277811 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.277723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-tmp-dir\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.277811 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.277753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8fr2\" (UniqueName: \"kubernetes.io/projected/f9cd3edd-e771-4821-a270-47c6d8bf6568-kube-api-access-k8fr2\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.277811 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.277777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-home\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.277973 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.277916 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9cd3edd-e771-4821-a270-47c6d8bf6568-tls-certs\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.278118 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.278094 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-model-cache\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.278202 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.278139 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-tmp-dir\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.278262 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.278211 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-home\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.279912 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.279892 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-dshm\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.280234 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.280216 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9cd3edd-e771-4821-a270-47c6d8bf6568-tls-certs\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.286172 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.286149 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8fr2\" (UniqueName: \"kubernetes.io/projected/f9cd3edd-e771-4821-a270-47c6d8bf6568-kube-api-access-k8fr2\") pod \"rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.404874 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.404848 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:09.538118 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.538080 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd"] Jul 23 01:12:09.542403 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:12:09.542371 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9cd3edd_e771_4821_a270_47c6d8bf6568.slice/crio-15e6b40ed51fdba3bb9738e9e1bb76b3ffa39e5ee470b4bbcfeef43d5a997dd1 WatchSource:0}: Error finding container 15e6b40ed51fdba3bb9738e9e1bb76b3ffa39e5ee470b4bbcfeef43d5a997dd1: Status 404 returned error can't find the container with id 15e6b40ed51fdba3bb9738e9e1bb76b3ffa39e5ee470b4bbcfeef43d5a997dd1 Jul 23 01:12:09.925813 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.925767 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" event={"ID":"f9cd3edd-e771-4821-a270-47c6d8bf6568","Type":"ContainerStarted","Data":"fe5b7faead929c75e84a161daa8412422e36805a506a8091ad8bd431c968f2cb"} Jul 23 01:12:09.925973 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.925824 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" event={"ID":"f9cd3edd-e771-4821-a270-47c6d8bf6568","Type":"ContainerStarted","Data":"15e6b40ed51fdba3bb9738e9e1bb76b3ffa39e5ee470b4bbcfeef43d5a997dd1"} Jul 23 01:12:09.943352 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:09.943307 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" podStartSLOduration=0.943292178 podStartE2EDuration="943.292178ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:12:09.942274264 +0000 UTC m=+2922.437942996" watchObservedRunningTime="2026-07-23 01:12:09.943292178 +0000 UTC m=+2922.438960890" Jul 23 01:12:19.405808 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:19.405767 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:19.405808 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:19.405812 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:19.418141 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:19.418114 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:19.994004 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:19.993978 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:12:28.434985 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:28.434948 2564 scope.go:117] "RemoveContainer" containerID="12f467f1d76137748c1965a0d506ebf5f86aa64109c6834a51428216fa02eab4" Jul 23 01:12:28.444410 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:28.444388 2564 scope.go:117] "RemoveContainer" containerID="108fc4c4c7ef2d0f4fb995bbb7d03ef2e39f4cedf99234a989f772d22cd82197" Jul 23 01:12:29.743222 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:29.743187 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:12:29.843892 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:29.843861 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:12:43.846293 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:43.846257 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:12:43.946724 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:43.946692 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:12:48.457177 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:48.457128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:12:49.165578 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:49.165544 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd"] Jul 23 01:12:49.182793 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:49.182764 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:12:50.172741 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:50.172707 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-140585907631280"] Jul 23 01:12:50.176320 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:50.176302 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907631280" Jul 23 01:12:50.178783 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:50.178765 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7rn92\"" Jul 23 01:12:50.184222 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:50.183962 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-140585907631280"] Jul 23 01:12:50.225611 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:50.225584 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtnr6\" (UniqueName: \"kubernetes.io/projected/d6dd1228-c833-487d-9ce7-0d34046552d3-kube-api-access-gtnr6\") pod \"test-curl-7567-140585907631280\" (UID: \"d6dd1228-c833-487d-9ce7-0d34046552d3\") " pod="opendatahub/test-curl-7567-140585907631280" Jul 23 01:12:50.326869 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:50.326841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtnr6\" (UniqueName: \"kubernetes.io/projected/d6dd1228-c833-487d-9ce7-0d34046552d3-kube-api-access-gtnr6\") pod \"test-curl-7567-140585907631280\" (UID: \"d6dd1228-c833-487d-9ce7-0d34046552d3\") " pod="opendatahub/test-curl-7567-140585907631280" Jul 23 01:12:50.334749 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:50.334726 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtnr6\" (UniqueName: \"kubernetes.io/projected/d6dd1228-c833-487d-9ce7-0d34046552d3-kube-api-access-gtnr6\") pod \"test-curl-7567-140585907631280\" (UID: \"d6dd1228-c833-487d-9ce7-0d34046552d3\") " pod="opendatahub/test-curl-7567-140585907631280" Jul 23 01:12:50.487225 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:50.487156 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907631280" Jul 23 01:12:50.617715 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:50.617694 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-140585907631280"] Jul 23 01:12:50.620106 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:12:50.620066 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6dd1228_c833_487d_9ce7_0d34046552d3.slice/crio-0c919f785da3d94446384b33928643a6ea62ce80c66db0902570e2aea4d02927 WatchSource:0}: Error finding container 0c919f785da3d94446384b33928643a6ea62ce80c66db0902570e2aea4d02927: Status 404 returned error can't find the container with id 0c919f785da3d94446384b33928643a6ea62ce80c66db0902570e2aea4d02927 Jul 23 01:12:50.621497 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:50.621479 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 01:12:51.164971 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:51.164926 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-140585907631280" event={"ID":"d6dd1228-c833-487d-9ce7-0d34046552d3","Type":"ContainerStarted","Data":"0c919f785da3d94446384b33928643a6ea62ce80c66db0902570e2aea4d02927"} Jul 23 01:12:51.739811 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:51.739778 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:12:51.840355 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:51.840328 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:12:52.171679 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:52.171645 2564 generic.go:356] "Generic (PLEG): container finished" podID="d6dd1228-c833-487d-9ce7-0d34046552d3" containerID="1e315d766fc767207ba26d69f07970d809539842cfd5d7a9664703c27bfe631e" exitCode=0 Jul 23 01:12:52.171868 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:52.171707 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-140585907631280" event={"ID":"d6dd1228-c833-487d-9ce7-0d34046552d3","Type":"ContainerDied","Data":"1e315d766fc767207ba26d69f07970d809539842cfd5d7a9664703c27bfe631e"} Jul 23 01:12:53.311558 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.311514 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907631280" Jul 23 01:12:53.332748 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.332724 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7567-140585907631280_d6dd1228-c833-487d-9ce7-0d34046552d3/test-curl-7567-140585907631280/0.log" Jul 23 01:12:53.351759 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.351730 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtnr6\" (UniqueName: \"kubernetes.io/projected/d6dd1228-c833-487d-9ce7-0d34046552d3-kube-api-access-gtnr6\") pod \"d6dd1228-c833-487d-9ce7-0d34046552d3\" (UID: \"d6dd1228-c833-487d-9ce7-0d34046552d3\") " Jul 23 01:12:53.353809 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.353777 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6dd1228-c833-487d-9ce7-0d34046552d3-kube-api-access-gtnr6" (OuterVolumeSpecName: "kube-api-access-gtnr6") pod "d6dd1228-c833-487d-9ce7-0d34046552d3" (UID: "d6dd1228-c833-487d-9ce7-0d34046552d3"). InnerVolumeSpecName "kube-api-access-gtnr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:12:53.359993 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.359971 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-140585907631280"] Jul 23 01:12:53.365627 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.365606 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-140585907631280"] Jul 23 01:12:53.453330 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.453268 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtnr6\" (UniqueName: \"kubernetes.io/projected/d6dd1228-c833-487d-9ce7-0d34046552d3-kube-api-access-gtnr6\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:12:53.493924 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.493900 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-140585907573168"] Jul 23 01:12:53.494332 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.494320 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6dd1228-c833-487d-9ce7-0d34046552d3" containerName="test-curl-7567-140585907631280" Jul 23 01:12:53.494378 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.494333 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6dd1228-c833-487d-9ce7-0d34046552d3" containerName="test-curl-7567-140585907631280" Jul 23 01:12:53.494418 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.494396 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6dd1228-c833-487d-9ce7-0d34046552d3" containerName="test-curl-7567-140585907631280" Jul 23 01:12:53.497503 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.497489 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907573168" Jul 23 01:12:53.504926 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.504902 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-140585907573168"] Jul 23 01:12:53.554063 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.554038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljl24\" (UniqueName: \"kubernetes.io/projected/b8a444dc-9404-46f8-90d4-cc18d3c1b5c3-kube-api-access-ljl24\") pod \"test-curl-7567-140585907573168\" (UID: \"b8a444dc-9404-46f8-90d4-cc18d3c1b5c3\") " pod="opendatahub/test-curl-7567-140585907573168" Jul 23 01:12:53.655038 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.655013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljl24\" (UniqueName: \"kubernetes.io/projected/b8a444dc-9404-46f8-90d4-cc18d3c1b5c3-kube-api-access-ljl24\") pod \"test-curl-7567-140585907573168\" (UID: \"b8a444dc-9404-46f8-90d4-cc18d3c1b5c3\") " pod="opendatahub/test-curl-7567-140585907573168" Jul 23 01:12:53.663288 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.663255 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljl24\" (UniqueName: \"kubernetes.io/projected/b8a444dc-9404-46f8-90d4-cc18d3c1b5c3-kube-api-access-ljl24\") pod \"test-curl-7567-140585907573168\" (UID: \"b8a444dc-9404-46f8-90d4-cc18d3c1b5c3\") " pod="opendatahub/test-curl-7567-140585907573168" Jul 23 01:12:53.808281 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.808219 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907573168" Jul 23 01:12:53.939507 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:53.939478 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-140585907573168"] Jul 23 01:12:53.940207 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:12:53.940184 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8a444dc_9404_46f8_90d4_cc18d3c1b5c3.slice/crio-e3137779ff300f669e7e4936ec010d259fab9795bd55c7a956c1548e63550de7 WatchSource:0}: Error finding container e3137779ff300f669e7e4936ec010d259fab9795bd55c7a956c1548e63550de7: Status 404 returned error can't find the container with id e3137779ff300f669e7e4936ec010d259fab9795bd55c7a956c1548e63550de7 Jul 23 01:12:54.074415 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:54.074340 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6dd1228-c833-487d-9ce7-0d34046552d3" path="/var/lib/kubelet/pods/d6dd1228-c833-487d-9ce7-0d34046552d3/volumes" Jul 23 01:12:54.184696 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:54.184664 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-140585907573168" event={"ID":"b8a444dc-9404-46f8-90d4-cc18d3c1b5c3","Type":"ContainerStarted","Data":"e3137779ff300f669e7e4936ec010d259fab9795bd55c7a956c1548e63550de7"} Jul 23 01:12:54.185969 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:54.185941 2564 scope.go:117] "RemoveContainer" containerID="1e315d766fc767207ba26d69f07970d809539842cfd5d7a9664703c27bfe631e" Jul 23 01:12:54.186077 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:54.185949 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907631280" Jul 23 01:12:55.193072 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:55.193037 2564 generic.go:356] "Generic (PLEG): container finished" podID="b8a444dc-9404-46f8-90d4-cc18d3c1b5c3" containerID="601ba9fa6886165ffba515ddd42ca7ec7ae1117c9b1db3a0697ae06b2fe23df5" exitCode=0 Jul 23 01:12:55.193512 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:55.193123 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-140585907573168" event={"ID":"b8a444dc-9404-46f8-90d4-cc18d3c1b5c3","Type":"ContainerDied","Data":"601ba9fa6886165ffba515ddd42ca7ec7ae1117c9b1db3a0697ae06b2fe23df5"} Jul 23 01:12:56.330958 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.330929 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907573168" Jul 23 01:12:56.350940 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.350917 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7567-140585907573168_b8a444dc-9404-46f8-90d4-cc18d3c1b5c3/test-curl-7567-140585907573168/0.log" Jul 23 01:12:56.374100 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.374072 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-140585907573168"] Jul 23 01:12:56.377915 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.377891 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-140585907573168"] Jul 23 01:12:56.380652 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.380634 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljl24\" (UniqueName: \"kubernetes.io/projected/b8a444dc-9404-46f8-90d4-cc18d3c1b5c3-kube-api-access-ljl24\") pod \"b8a444dc-9404-46f8-90d4-cc18d3c1b5c3\" (UID: \"b8a444dc-9404-46f8-90d4-cc18d3c1b5c3\") " Jul 23 01:12:56.382608 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.382586 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a444dc-9404-46f8-90d4-cc18d3c1b5c3-kube-api-access-ljl24" (OuterVolumeSpecName: "kube-api-access-ljl24") pod "b8a444dc-9404-46f8-90d4-cc18d3c1b5c3" (UID: "b8a444dc-9404-46f8-90d4-cc18d3c1b5c3"). InnerVolumeSpecName "kube-api-access-ljl24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:12:56.481734 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.481669 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljl24\" (UniqueName: \"kubernetes.io/projected/b8a444dc-9404-46f8-90d4-cc18d3c1b5c3-kube-api-access-ljl24\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:12:56.538284 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.538253 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-140585907700624"] Jul 23 01:12:56.538714 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.538699 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a444dc-9404-46f8-90d4-cc18d3c1b5c3" containerName="test-curl-7567-140585907573168" Jul 23 01:12:56.538714 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.538715 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a444dc-9404-46f8-90d4-cc18d3c1b5c3" containerName="test-curl-7567-140585907573168" Jul 23 01:12:56.538820 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.538787 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8a444dc-9404-46f8-90d4-cc18d3c1b5c3" containerName="test-curl-7567-140585907573168" Jul 23 01:12:56.542048 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.541791 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907700624" Jul 23 01:12:56.545580 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.545558 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-140585907700624"] Jul 23 01:12:56.683172 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.683145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv5ph\" (UniqueName: \"kubernetes.io/projected/2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4-kube-api-access-bv5ph\") pod \"test-curl-7567-140585907700624\" (UID: \"2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4\") " pod="opendatahub/test-curl-7567-140585907700624" Jul 23 01:12:56.784197 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.784127 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv5ph\" (UniqueName: \"kubernetes.io/projected/2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4-kube-api-access-bv5ph\") pod \"test-curl-7567-140585907700624\" (UID: \"2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4\") " pod="opendatahub/test-curl-7567-140585907700624" Jul 23 01:12:56.793063 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.793036 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv5ph\" (UniqueName: \"kubernetes.io/projected/2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4-kube-api-access-bv5ph\") pod \"test-curl-7567-140585907700624\" (UID: \"2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4\") " pod="opendatahub/test-curl-7567-140585907700624" Jul 23 01:12:56.853079 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.853050 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907700624" Jul 23 01:12:56.984950 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:56.984926 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-140585907700624"] Jul 23 01:12:56.986184 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:12:56.986156 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d681b0a_3b9a_4f1a_85c8_22af7eb6dfc4.slice/crio-ca5e0dd4b427f31bf15ac6a6c8c695a373c9683a5f17a0ccd1d400d96b4d2c7b WatchSource:0}: Error finding container ca5e0dd4b427f31bf15ac6a6c8c695a373c9683a5f17a0ccd1d400d96b4d2c7b: Status 404 returned error can't find the container with id ca5e0dd4b427f31bf15ac6a6c8c695a373c9683a5f17a0ccd1d400d96b4d2c7b Jul 23 01:12:57.215302 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:57.215273 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907573168" Jul 23 01:12:57.215480 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:57.215269 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3137779ff300f669e7e4936ec010d259fab9795bd55c7a956c1548e63550de7" Jul 23 01:12:57.216772 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:57.216743 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-140585907700624" event={"ID":"2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4","Type":"ContainerStarted","Data":"ca5e0dd4b427f31bf15ac6a6c8c695a373c9683a5f17a0ccd1d400d96b4d2c7b"} Jul 23 01:12:57.226688 ip-10-0-139-1 kubenswrapper[2564]: E0723 01:12:57.226662 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7567-140585907573168\" is forbidden: User \"system:node:ip-10-0-139-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-1.ec2.internal' and this object" podUID="b8a444dc-9404-46f8-90d4-cc18d3c1b5c3" pod="opendatahub/test-curl-7567-140585907573168" Jul 23 01:12:58.073636 ip-10-0-139-1 kubenswrapper[2564]: E0723 01:12:58.073601 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7567-140585907573168\" is forbidden: User \"system:node:ip-10-0-139-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-1.ec2.internal' and this object" podUID="b8a444dc-9404-46f8-90d4-cc18d3c1b5c3" pod="opendatahub/test-curl-7567-140585907573168" Jul 23 01:12:58.074702 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:58.074682 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8a444dc-9404-46f8-90d4-cc18d3c1b5c3" path="/var/lib/kubelet/pods/b8a444dc-9404-46f8-90d4-cc18d3c1b5c3/volumes" Jul 23 01:12:58.223933 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:58.223902 2564 generic.go:356] "Generic (PLEG): container finished" podID="2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4" containerID="7942efdee75fbf4179d019b3a5476194bf373104152b7931254d974628063931" exitCode=0 Jul 23 01:12:58.224073 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:58.223983 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-140585907700624" event={"ID":"2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4","Type":"ContainerDied","Data":"7942efdee75fbf4179d019b3a5476194bf373104152b7931254d974628063931"} Jul 23 01:12:59.369232 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.369209 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907700624" Jul 23 01:12:59.387750 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.387727 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7567-140585907700624_2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4/test-curl-7567-140585907700624/0.log" Jul 23 01:12:59.410803 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.410770 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-140585907700624"] Jul 23 01:12:59.413044 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.413023 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-140585907700624"] Jul 23 01:12:59.507115 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.507050 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bv5ph\" (UniqueName: \"kubernetes.io/projected/2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4-kube-api-access-bv5ph\") pod \"2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4\" (UID: \"2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4\") " Jul 23 01:12:59.509083 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.509056 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4-kube-api-access-bv5ph" (OuterVolumeSpecName: "kube-api-access-bv5ph") pod "2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4" (UID: "2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4"). InnerVolumeSpecName "kube-api-access-bv5ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:12:59.544514 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.544491 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-140585907700624"] Jul 23 01:12:59.544952 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.544937 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4" containerName="test-curl-7567-140585907700624" Jul 23 01:12:59.545002 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.544954 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4" containerName="test-curl-7567-140585907700624" Jul 23 01:12:59.545038 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.545018 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4" containerName="test-curl-7567-140585907700624" Jul 23 01:12:59.548103 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.548088 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907700624" Jul 23 01:12:59.552329 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.552305 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-140585907700624"] Jul 23 01:12:59.608134 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.608108 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n65m\" (UniqueName: \"kubernetes.io/projected/550f50b4-311a-47cb-b0ac-01e77c8e484a-kube-api-access-5n65m\") pod \"test-curl-7567-140585907700624\" (UID: \"550f50b4-311a-47cb-b0ac-01e77c8e484a\") " pod="opendatahub/test-curl-7567-140585907700624" Jul 23 01:12:59.608240 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.608164 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv5ph\" (UniqueName: \"kubernetes.io/projected/2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4-kube-api-access-bv5ph\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:12:59.708561 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.708511 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n65m\" (UniqueName: \"kubernetes.io/projected/550f50b4-311a-47cb-b0ac-01e77c8e484a-kube-api-access-5n65m\") pod \"test-curl-7567-140585907700624\" (UID: \"550f50b4-311a-47cb-b0ac-01e77c8e484a\") " pod="opendatahub/test-curl-7567-140585907700624" Jul 23 01:12:59.716609 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.716593 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n65m\" (UniqueName: \"kubernetes.io/projected/550f50b4-311a-47cb-b0ac-01e77c8e484a-kube-api-access-5n65m\") pod \"test-curl-7567-140585907700624\" (UID: \"550f50b4-311a-47cb-b0ac-01e77c8e484a\") " pod="opendatahub/test-curl-7567-140585907700624" Jul 23 01:12:59.858518 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.858446 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907700624" Jul 23 01:12:59.992001 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:12:59.991978 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-140585907700624"] Jul 23 01:12:59.993342 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:12:59.993313 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod550f50b4_311a_47cb_b0ac_01e77c8e484a.slice/crio-5c3f5bb66986095fa84333904291c5e5c7a97ddc008150df15cdf0770634898c WatchSource:0}: Error finding container 5c3f5bb66986095fa84333904291c5e5c7a97ddc008150df15cdf0770634898c: Status 404 returned error can't find the container with id 5c3f5bb66986095fa84333904291c5e5c7a97ddc008150df15cdf0770634898c Jul 23 01:13:00.073384 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:00.073354 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4" path="/var/lib/kubelet/pods/2d681b0a-3b9a-4f1a-85c8-22af7eb6dfc4/volumes" Jul 23 01:13:00.236975 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:00.236940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-140585907700624" event={"ID":"550f50b4-311a-47cb-b0ac-01e77c8e484a","Type":"ContainerStarted","Data":"5c3f5bb66986095fa84333904291c5e5c7a97ddc008150df15cdf0770634898c"} Jul 23 01:13:00.238269 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:00.238252 2564 scope.go:117] "RemoveContainer" containerID="7942efdee75fbf4179d019b3a5476194bf373104152b7931254d974628063931" Jul 23 01:13:00.238413 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:00.238273 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907700624" Jul 23 01:13:01.246233 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:01.246195 2564 generic.go:356] "Generic (PLEG): container finished" podID="550f50b4-311a-47cb-b0ac-01e77c8e484a" containerID="53e83961542e33b52fa6b32f3b5600356a44010af2dcf091be6919af70204c44" exitCode=0 Jul 23 01:13:01.246756 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:01.246291 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-140585907700624" event={"ID":"550f50b4-311a-47cb-b0ac-01e77c8e484a","Type":"ContainerDied","Data":"53e83961542e33b52fa6b32f3b5600356a44010af2dcf091be6919af70204c44"} Jul 23 01:13:02.390501 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.390479 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907700624" Jul 23 01:13:02.411327 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.411291 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7567-140585907700624_550f50b4-311a-47cb-b0ac-01e77c8e484a/test-curl-7567-140585907700624/0.log" Jul 23 01:13:02.437457 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.437434 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-140585907700624"] Jul 23 01:13:02.441380 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.441360 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-140585907700624"] Jul 23 01:13:02.532290 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.532237 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n65m\" (UniqueName: \"kubernetes.io/projected/550f50b4-311a-47cb-b0ac-01e77c8e484a-kube-api-access-5n65m\") pod \"550f50b4-311a-47cb-b0ac-01e77c8e484a\" (UID: \"550f50b4-311a-47cb-b0ac-01e77c8e484a\") " Jul 23 01:13:02.534232 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.534211 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550f50b4-311a-47cb-b0ac-01e77c8e484a-kube-api-access-5n65m" (OuterVolumeSpecName: "kube-api-access-5n65m") pod "550f50b4-311a-47cb-b0ac-01e77c8e484a" (UID: "550f50b4-311a-47cb-b0ac-01e77c8e484a"). InnerVolumeSpecName "kube-api-access-5n65m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:13:02.560794 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.560769 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-140585907701632"] Jul 23 01:13:02.561169 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.561156 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550f50b4-311a-47cb-b0ac-01e77c8e484a" containerName="test-curl-7567-140585907700624" Jul 23 01:13:02.561217 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.561170 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="550f50b4-311a-47cb-b0ac-01e77c8e484a" containerName="test-curl-7567-140585907700624" Jul 23 01:13:02.561259 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.561240 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="550f50b4-311a-47cb-b0ac-01e77c8e484a" containerName="test-curl-7567-140585907700624" Jul 23 01:13:02.564175 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.564161 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907701632" Jul 23 01:13:02.569310 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.569287 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-140585907701632"] Jul 23 01:13:02.633320 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.633297 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hlvb\" (UniqueName: \"kubernetes.io/projected/69f193be-b2cc-41c1-8f02-7d6dac99b80c-kube-api-access-8hlvb\") pod \"test-curl-7567-140585907701632\" (UID: \"69f193be-b2cc-41c1-8f02-7d6dac99b80c\") " pod="opendatahub/test-curl-7567-140585907701632" Jul 23 01:13:02.633414 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.633401 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n65m\" (UniqueName: \"kubernetes.io/projected/550f50b4-311a-47cb-b0ac-01e77c8e484a-kube-api-access-5n65m\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:13:02.733924 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.733898 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hlvb\" (UniqueName: \"kubernetes.io/projected/69f193be-b2cc-41c1-8f02-7d6dac99b80c-kube-api-access-8hlvb\") pod \"test-curl-7567-140585907701632\" (UID: \"69f193be-b2cc-41c1-8f02-7d6dac99b80c\") " pod="opendatahub/test-curl-7567-140585907701632" Jul 23 01:13:02.751167 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.751144 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hlvb\" (UniqueName: \"kubernetes.io/projected/69f193be-b2cc-41c1-8f02-7d6dac99b80c-kube-api-access-8hlvb\") pod \"test-curl-7567-140585907701632\" (UID: \"69f193be-b2cc-41c1-8f02-7d6dac99b80c\") " pod="opendatahub/test-curl-7567-140585907701632" Jul 23 01:13:02.874599 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:02.874546 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907701632" Jul 23 01:13:03.004903 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:03.004865 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-140585907701632"] Jul 23 01:13:03.006547 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:13:03.006506 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69f193be_b2cc_41c1_8f02_7d6dac99b80c.slice/crio-8e927c6d25693faee988de12bd08fd32468d5f2a9d43dbcb5b5a82380f370ebf WatchSource:0}: Error finding container 8e927c6d25693faee988de12bd08fd32468d5f2a9d43dbcb5b5a82380f370ebf: Status 404 returned error can't find the container with id 8e927c6d25693faee988de12bd08fd32468d5f2a9d43dbcb5b5a82380f370ebf Jul 23 01:13:03.260014 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:03.259985 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907700624" Jul 23 01:13:03.260164 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:03.260018 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c3f5bb66986095fa84333904291c5e5c7a97ddc008150df15cdf0770634898c" Jul 23 01:13:03.261174 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:03.261148 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-140585907701632" event={"ID":"69f193be-b2cc-41c1-8f02-7d6dac99b80c","Type":"ContainerStarted","Data":"8e927c6d25693faee988de12bd08fd32468d5f2a9d43dbcb5b5a82380f370ebf"} Jul 23 01:13:03.271029 ip-10-0-139-1 kubenswrapper[2564]: E0723 01:13:03.271006 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7567-140585907700624\" is forbidden: User \"system:node:ip-10-0-139-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-1.ec2.internal' and this object" podUID="550f50b4-311a-47cb-b0ac-01e77c8e484a" pod="opendatahub/test-curl-7567-140585907700624" Jul 23 01:13:04.073310 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.073274 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="550f50b4-311a-47cb-b0ac-01e77c8e484a" path="/var/lib/kubelet/pods/550f50b4-311a-47cb-b0ac-01e77c8e484a/volumes" Jul 23 01:13:04.186466 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.186423 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" podUID="f9cd3edd-e771-4821-a270-47c6d8bf6568" containerName="main" containerID="cri-o://fe5b7faead929c75e84a161daa8412422e36805a506a8091ad8bd431c968f2cb" gracePeriod=45 Jul 23 01:13:04.269005 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.268971 2564 generic.go:356] "Generic (PLEG): container finished" podID="69f193be-b2cc-41c1-8f02-7d6dac99b80c" containerID="8461f0ce5e35c2d7e266877a2dfc9bbe4165a8b82cd753636602c4a57e9f55fc" exitCode=0 Jul 23 01:13:04.269145 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.269058 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-140585907701632" event={"ID":"69f193be-b2cc-41c1-8f02-7d6dac99b80c","Type":"ContainerDied","Data":"8461f0ce5e35c2d7e266877a2dfc9bbe4165a8b82cd753636602c4a57e9f55fc"} Jul 23 01:13:04.438291 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.438268 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:13:04.551248 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.551219 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9cd3edd-e771-4821-a270-47c6d8bf6568-tls-certs\") pod \"f9cd3edd-e771-4821-a270-47c6d8bf6568\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " Jul 23 01:13:04.551381 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.551275 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-model-cache\") pod \"f9cd3edd-e771-4821-a270-47c6d8bf6568\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " Jul 23 01:13:04.551381 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.551363 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8fr2\" (UniqueName: \"kubernetes.io/projected/f9cd3edd-e771-4821-a270-47c6d8bf6568-kube-api-access-k8fr2\") pod \"f9cd3edd-e771-4821-a270-47c6d8bf6568\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " Jul 23 01:13:04.551512 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.551399 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-home\") pod \"f9cd3edd-e771-4821-a270-47c6d8bf6568\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " Jul 23 01:13:04.551512 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.551443 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-tmp-dir\") pod \"f9cd3edd-e771-4821-a270-47c6d8bf6568\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " Jul 23 01:13:04.551512 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.551474 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-dshm\") pod \"f9cd3edd-e771-4821-a270-47c6d8bf6568\" (UID: \"f9cd3edd-e771-4821-a270-47c6d8bf6568\") " Jul 23 01:13:04.551705 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.551586 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9cd3edd-e771-4821-a270-47c6d8bf6568" (UID: "f9cd3edd-e771-4821-a270-47c6d8bf6568"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:13:04.551705 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.551642 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-home" (OuterVolumeSpecName: "home") pod "f9cd3edd-e771-4821-a270-47c6d8bf6568" (UID: "f9cd3edd-e771-4821-a270-47c6d8bf6568"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:13:04.551818 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.551787 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-home\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:13:04.551818 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.551781 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9cd3edd-e771-4821-a270-47c6d8bf6568" (UID: "f9cd3edd-e771-4821-a270-47c6d8bf6568"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:13:04.551818 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.551799 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-model-cache\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:13:04.553458 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.553428 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9cd3edd-e771-4821-a270-47c6d8bf6568-kube-api-access-k8fr2" (OuterVolumeSpecName: "kube-api-access-k8fr2") pod "f9cd3edd-e771-4821-a270-47c6d8bf6568" (UID: "f9cd3edd-e771-4821-a270-47c6d8bf6568"). InnerVolumeSpecName "kube-api-access-k8fr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:13:04.553589 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.553461 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9cd3edd-e771-4821-a270-47c6d8bf6568-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9cd3edd-e771-4821-a270-47c6d8bf6568" (UID: "f9cd3edd-e771-4821-a270-47c6d8bf6568"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:13:04.553589 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.553476 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-dshm" (OuterVolumeSpecName: "dshm") pod "f9cd3edd-e771-4821-a270-47c6d8bf6568" (UID: "f9cd3edd-e771-4821-a270-47c6d8bf6568"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 01:13:04.652579 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.652558 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8fr2\" (UniqueName: \"kubernetes.io/projected/f9cd3edd-e771-4821-a270-47c6d8bf6568-kube-api-access-k8fr2\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:13:04.652679 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.652579 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-tmp-dir\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:13:04.652679 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.652591 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9cd3edd-e771-4821-a270-47c6d8bf6568-dshm\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:13:04.652679 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:04.652598 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9cd3edd-e771-4821-a270-47c6d8bf6568-tls-certs\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:13:05.276456 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.276425 2564 generic.go:356] "Generic (PLEG): container finished" podID="f9cd3edd-e771-4821-a270-47c6d8bf6568" containerID="fe5b7faead929c75e84a161daa8412422e36805a506a8091ad8bd431c968f2cb" exitCode=0 Jul 23 01:13:05.276872 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.276471 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" Jul 23 01:13:05.276872 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.276514 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" event={"ID":"f9cd3edd-e771-4821-a270-47c6d8bf6568","Type":"ContainerDied","Data":"fe5b7faead929c75e84a161daa8412422e36805a506a8091ad8bd431c968f2cb"} Jul 23 01:13:05.276872 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.276580 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd" event={"ID":"f9cd3edd-e771-4821-a270-47c6d8bf6568","Type":"ContainerDied","Data":"15e6b40ed51fdba3bb9738e9e1bb76b3ffa39e5ee470b4bbcfeef43d5a997dd1"} Jul 23 01:13:05.276872 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.276603 2564 scope.go:117] "RemoveContainer" containerID="fe5b7faead929c75e84a161daa8412422e36805a506a8091ad8bd431c968f2cb" Jul 23 01:13:05.287632 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.287614 2564 scope.go:117] "RemoveContainer" containerID="fe5b7faead929c75e84a161daa8412422e36805a506a8091ad8bd431c968f2cb" Jul 23 01:13:05.287907 ip-10-0-139-1 kubenswrapper[2564]: E0723 01:13:05.287889 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5b7faead929c75e84a161daa8412422e36805a506a8091ad8bd431c968f2cb\": container with ID starting with fe5b7faead929c75e84a161daa8412422e36805a506a8091ad8bd431c968f2cb not found: ID does not exist" containerID="fe5b7faead929c75e84a161daa8412422e36805a506a8091ad8bd431c968f2cb" Jul 23 01:13:05.287950 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.287915 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5b7faead929c75e84a161daa8412422e36805a506a8091ad8bd431c968f2cb"} err="failed to get container status \"fe5b7faead929c75e84a161daa8412422e36805a506a8091ad8bd431c968f2cb\": rpc error: code = NotFound desc = could not find container \"fe5b7faead929c75e84a161daa8412422e36805a506a8091ad8bd431c968f2cb\": container with ID starting with fe5b7faead929c75e84a161daa8412422e36805a506a8091ad8bd431c968f2cb not found: ID does not exist" Jul 23 01:13:05.308822 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.308795 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd"] Jul 23 01:13:05.315841 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.315813 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d6869a/rate-test-model-d6869a-kserve-5cdfd5d4c-d8jpd"] Jul 23 01:13:05.413336 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.413315 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907701632" Jul 23 01:13:05.431798 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.431765 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7567-140585907701632_69f193be-b2cc-41c1-8f02-7d6dac99b80c/test-curl-7567-140585907701632/0.log" Jul 23 01:13:05.454435 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.454410 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-140585907701632"] Jul 23 01:13:05.456690 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.456670 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-140585907701632"] Jul 23 01:13:05.558501 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.558434 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hlvb\" (UniqueName: \"kubernetes.io/projected/69f193be-b2cc-41c1-8f02-7d6dac99b80c-kube-api-access-8hlvb\") pod \"69f193be-b2cc-41c1-8f02-7d6dac99b80c\" (UID: \"69f193be-b2cc-41c1-8f02-7d6dac99b80c\") " Jul 23 01:13:05.560413 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.560391 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f193be-b2cc-41c1-8f02-7d6dac99b80c-kube-api-access-8hlvb" (OuterVolumeSpecName: "kube-api-access-8hlvb") pod "69f193be-b2cc-41c1-8f02-7d6dac99b80c" (UID: "69f193be-b2cc-41c1-8f02-7d6dac99b80c"). InnerVolumeSpecName "kube-api-access-8hlvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:13:05.585902 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.585877 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7567-140585907701632"] Jul 23 01:13:05.586303 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.586290 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69f193be-b2cc-41c1-8f02-7d6dac99b80c" containerName="test-curl-7567-140585907701632" Jul 23 01:13:05.586347 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.586304 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="69f193be-b2cc-41c1-8f02-7d6dac99b80c" containerName="test-curl-7567-140585907701632" Jul 23 01:13:05.586347 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.586324 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9cd3edd-e771-4821-a270-47c6d8bf6568" containerName="main" Jul 23 01:13:05.586347 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.586331 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9cd3edd-e771-4821-a270-47c6d8bf6568" containerName="main" Jul 23 01:13:05.586444 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.586392 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="69f193be-b2cc-41c1-8f02-7d6dac99b80c" containerName="test-curl-7567-140585907701632" Jul 23 01:13:05.586444 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.586403 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9cd3edd-e771-4821-a270-47c6d8bf6568" containerName="main" Jul 23 01:13:05.589345 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.589331 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907701632" Jul 23 01:13:05.595821 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.595787 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-140585907701632"] Jul 23 01:13:05.659919 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.659896 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnp4b\" (UniqueName: \"kubernetes.io/projected/c93e57f3-ba0a-4c62-a71b-636b8c8c1374-kube-api-access-hnp4b\") pod \"test-curl-7567-140585907701632\" (UID: \"c93e57f3-ba0a-4c62-a71b-636b8c8c1374\") " pod="opendatahub/test-curl-7567-140585907701632" Jul 23 01:13:05.660040 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.659942 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hlvb\" (UniqueName: \"kubernetes.io/projected/69f193be-b2cc-41c1-8f02-7d6dac99b80c-kube-api-access-8hlvb\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:13:05.760895 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.760865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnp4b\" (UniqueName: \"kubernetes.io/projected/c93e57f3-ba0a-4c62-a71b-636b8c8c1374-kube-api-access-hnp4b\") pod \"test-curl-7567-140585907701632\" (UID: \"c93e57f3-ba0a-4c62-a71b-636b8c8c1374\") " pod="opendatahub/test-curl-7567-140585907701632" Jul 23 01:13:05.769878 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.769849 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnp4b\" (UniqueName: \"kubernetes.io/projected/c93e57f3-ba0a-4c62-a71b-636b8c8c1374-kube-api-access-hnp4b\") pod \"test-curl-7567-140585907701632\" (UID: \"c93e57f3-ba0a-4c62-a71b-636b8c8c1374\") " pod="opendatahub/test-curl-7567-140585907701632" Jul 23 01:13:05.899585 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:05.899563 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907701632" Jul 23 01:13:06.032887 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:06.032860 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7567-140585907701632"] Jul 23 01:13:06.034312 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:13:06.034291 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc93e57f3_ba0a_4c62_a71b_636b8c8c1374.slice/crio-89949b25c21e607f507d1058af8232b71cb4da7e68d347a0c7526b03f766c6d3 WatchSource:0}: Error finding container 89949b25c21e607f507d1058af8232b71cb4da7e68d347a0c7526b03f766c6d3: Status 404 returned error can't find the container with id 89949b25c21e607f507d1058af8232b71cb4da7e68d347a0c7526b03f766c6d3 Jul 23 01:13:06.076670 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:06.076648 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69f193be-b2cc-41c1-8f02-7d6dac99b80c" path="/var/lib/kubelet/pods/69f193be-b2cc-41c1-8f02-7d6dac99b80c/volumes" Jul 23 01:13:06.077045 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:06.077029 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9cd3edd-e771-4821-a270-47c6d8bf6568" path="/var/lib/kubelet/pods/f9cd3edd-e771-4821-a270-47c6d8bf6568/volumes" Jul 23 01:13:06.284108 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:06.284041 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907701632" Jul 23 01:13:06.284108 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:06.284094 2564 scope.go:117] "RemoveContainer" containerID="8461f0ce5e35c2d7e266877a2dfc9bbe4165a8b82cd753636602c4a57e9f55fc" Jul 23 01:13:06.285269 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:06.285244 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-140585907701632" event={"ID":"c93e57f3-ba0a-4c62-a71b-636b8c8c1374","Type":"ContainerStarted","Data":"89949b25c21e607f507d1058af8232b71cb4da7e68d347a0c7526b03f766c6d3"} Jul 23 01:13:07.293641 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:07.293611 2564 generic.go:356] "Generic (PLEG): container finished" podID="c93e57f3-ba0a-4c62-a71b-636b8c8c1374" containerID="fe3dd6ca5655c17efa32b6e71f28ef6b0d8585e6c60b89944bf187b40d89db11" exitCode=0 Jul 23 01:13:07.294008 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:07.293667 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7567-140585907701632" event={"ID":"c93e57f3-ba0a-4c62-a71b-636b8c8c1374","Type":"ContainerDied","Data":"fe3dd6ca5655c17efa32b6e71f28ef6b0d8585e6c60b89944bf187b40d89db11"} Jul 23 01:13:08.432142 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.432118 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907701632" Jul 23 01:13:08.450898 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.450875 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7567-140585907701632_c93e57f3-ba0a-4c62-a71b-636b8c8c1374/test-curl-7567-140585907701632/0.log" Jul 23 01:13:08.474567 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.474541 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7567-140585907701632"] Jul 23 01:13:08.478773 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.478749 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7567-140585907701632"] Jul 23 01:13:08.585930 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.585869 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnp4b\" (UniqueName: \"kubernetes.io/projected/c93e57f3-ba0a-4c62-a71b-636b8c8c1374-kube-api-access-hnp4b\") pod \"c93e57f3-ba0a-4c62-a71b-636b8c8c1374\" (UID: \"c93e57f3-ba0a-4c62-a71b-636b8c8c1374\") " Jul 23 01:13:08.587878 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.587859 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93e57f3-ba0a-4c62-a71b-636b8c8c1374-kube-api-access-hnp4b" (OuterVolumeSpecName: "kube-api-access-hnp4b") pod "c93e57f3-ba0a-4c62-a71b-636b8c8c1374" (UID: "c93e57f3-ba0a-4c62-a71b-636b8c8c1374"). InnerVolumeSpecName "kube-api-access-hnp4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:13:08.664600 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.664571 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j"] Jul 23 01:13:08.664977 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.664965 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c93e57f3-ba0a-4c62-a71b-636b8c8c1374" containerName="test-curl-7567-140585907701632" Jul 23 01:13:08.665026 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.664979 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c93e57f3-ba0a-4c62-a71b-636b8c8c1374" containerName="test-curl-7567-140585907701632" Jul 23 01:13:08.665062 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.665046 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c93e57f3-ba0a-4c62-a71b-636b8c8c1374" containerName="test-curl-7567-140585907701632" Jul 23 01:13:08.667999 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.667984 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j" Jul 23 01:13:08.670365 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.670349 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zd6c\"" Jul 23 01:13:08.676407 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.675867 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j"] Jul 23 01:13:08.687093 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.687073 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnp4b\" (UniqueName: \"kubernetes.io/projected/c93e57f3-ba0a-4c62-a71b-636b8c8c1374-kube-api-access-hnp4b\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:13:08.788080 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.788053 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0710c809-44a1-4f85-9419-c37a0b11f59e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j\" (UID: \"0710c809-44a1-4f85-9419-c37a0b11f59e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j" Jul 23 01:13:08.888684 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.888643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0710c809-44a1-4f85-9419-c37a0b11f59e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j\" (UID: \"0710c809-44a1-4f85-9419-c37a0b11f59e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j" Jul 23 01:13:08.889587 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.889568 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0710c809-44a1-4f85-9419-c37a0b11f59e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j\" (UID: \"0710c809-44a1-4f85-9419-c37a0b11f59e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j" Jul 23 01:13:08.980617 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:08.980590 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j" Jul 23 01:13:09.114466 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:09.114441 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j"] Jul 23 01:13:09.116410 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:13:09.116385 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0710c809_44a1_4f85_9419_c37a0b11f59e.slice/crio-89e6680d588ccd50edd817879aa4b5870a170c7d77977d26a79b8521b70e2746 WatchSource:0}: Error finding container 89e6680d588ccd50edd817879aa4b5870a170c7d77977d26a79b8521b70e2746: Status 404 returned error can't find the container with id 89e6680d588ccd50edd817879aa4b5870a170c7d77977d26a79b8521b70e2746 Jul 23 01:13:09.271964 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:09.271932 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:09.306492 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:09.306464 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j" event={"ID":"0710c809-44a1-4f85-9419-c37a0b11f59e","Type":"ContainerStarted","Data":"89e6680d588ccd50edd817879aa4b5870a170c7d77977d26a79b8521b70e2746"} Jul 23 01:13:09.307653 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:09.307632 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89949b25c21e607f507d1058af8232b71cb4da7e68d347a0c7526b03f766c6d3" Jul 23 01:13:09.307743 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:09.307663 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7567-140585907701632" Jul 23 01:13:09.317368 ip-10-0-139-1 kubenswrapper[2564]: E0723 01:13:09.317340 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7567-140585907701632\" is forbidden: User \"system:node:ip-10-0-139-1.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-1.ec2.internal' and this object" podUID="c93e57f3-ba0a-4c62-a71b-636b8c8c1374" pod="opendatahub/test-curl-7567-140585907701632" Jul 23 01:13:09.372446 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:09.372408 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:10.073310 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:10.073233 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c93e57f3-ba0a-4c62-a71b-636b8c8c1374" path="/var/lib/kubelet/pods/c93e57f3-ba0a-4c62-a71b-636b8c8c1374/volumes" Jul 23 01:13:10.314589 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:10.314554 2564 generic.go:356] "Generic (PLEG): container finished" podID="0710c809-44a1-4f85-9419-c37a0b11f59e" containerID="b81a7aa0ab90c70057615a08a2a9cb41008a0bcbd4664f8994f3f780737ceef0" exitCode=0 Jul 23 01:13:10.314768 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:10.314638 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j" event={"ID":"0710c809-44a1-4f85-9419-c37a0b11f59e","Type":"ContainerDied","Data":"b81a7aa0ab90c70057615a08a2a9cb41008a0bcbd4664f8994f3f780737ceef0"} Jul 23 01:13:11.458494 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:11.458468 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j" Jul 23 01:13:11.613515 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:11.613446 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0710c809-44a1-4f85-9419-c37a0b11f59e-maas-api-service-ca\") pod \"0710c809-44a1-4f85-9419-c37a0b11f59e\" (UID: \"0710c809-44a1-4f85-9419-c37a0b11f59e\") " Jul 23 01:13:11.613755 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:11.613732 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0710c809-44a1-4f85-9419-c37a0b11f59e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0710c809-44a1-4f85-9419-c37a0b11f59e" (UID: "0710c809-44a1-4f85-9419-c37a0b11f59e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:13:11.714638 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:11.714612 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0710c809-44a1-4f85-9419-c37a0b11f59e-maas-api-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:13:12.327965 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:12.327940 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j" Jul 23 01:13:12.327965 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:12.327955 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j" event={"ID":"0710c809-44a1-4f85-9419-c37a0b11f59e","Type":"ContainerDied","Data":"89e6680d588ccd50edd817879aa4b5870a170c7d77977d26a79b8521b70e2746"} Jul 23 01:13:12.328176 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:12.328003 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89e6680d588ccd50edd817879aa4b5870a170c7d77977d26a79b8521b70e2746" Jul 23 01:13:19.103665 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:19.103635 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc"] Jul 23 01:13:19.104119 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:19.103826 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" podUID="6e9ccc9b-9902-4c90-b9de-c06204a36322" containerName="maas-api" containerID="cri-o://ddde4535d5b481c26e6c4c17f816d6f64f8b20bd86917129c753d3da50ea57e2" gracePeriod=30 Jul 23 01:13:19.128065 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:19.128034 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:19.288324 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:19.288282 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" podUID="6e9ccc9b-9902-4c90-b9de-c06204a36322" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.63:8443/health\": dial tcp 10.134.0.63:8443: connect: connection refused" Jul 23 01:13:19.574099 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:19.574070 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" Jul 23 01:13:19.682749 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:19.682721 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb2nz\" (UniqueName: \"kubernetes.io/projected/6e9ccc9b-9902-4c90-b9de-c06204a36322-kube-api-access-lb2nz\") pod \"6e9ccc9b-9902-4c90-b9de-c06204a36322\" (UID: \"6e9ccc9b-9902-4c90-b9de-c06204a36322\") " Jul 23 01:13:19.682905 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:19.682761 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9ccc9b-9902-4c90-b9de-c06204a36322-maas-api-tls\") pod \"6e9ccc9b-9902-4c90-b9de-c06204a36322\" (UID: \"6e9ccc9b-9902-4c90-b9de-c06204a36322\") " Jul 23 01:13:19.682905 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:19.682874 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ccc9b-9902-4c90-b9de-c06204a36322-openshift-service-ca\") pod \"6e9ccc9b-9902-4c90-b9de-c06204a36322\" (UID: \"6e9ccc9b-9902-4c90-b9de-c06204a36322\") " Jul 23 01:13:19.683204 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:19.683180 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9ccc9b-9902-4c90-b9de-c06204a36322-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6e9ccc9b-9902-4c90-b9de-c06204a36322" (UID: "6e9ccc9b-9902-4c90-b9de-c06204a36322"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:13:19.684855 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:19.684822 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e9ccc9b-9902-4c90-b9de-c06204a36322-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6e9ccc9b-9902-4c90-b9de-c06204a36322" (UID: "6e9ccc9b-9902-4c90-b9de-c06204a36322"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 01:13:19.684979 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:19.684870 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9ccc9b-9902-4c90-b9de-c06204a36322-kube-api-access-lb2nz" (OuterVolumeSpecName: "kube-api-access-lb2nz") pod "6e9ccc9b-9902-4c90-b9de-c06204a36322" (UID: "6e9ccc9b-9902-4c90-b9de-c06204a36322"). InnerVolumeSpecName "kube-api-access-lb2nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 01:13:19.784629 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:19.784602 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ccc9b-9902-4c90-b9de-c06204a36322-openshift-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:13:19.784629 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:19.784624 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb2nz\" (UniqueName: \"kubernetes.io/projected/6e9ccc9b-9902-4c90-b9de-c06204a36322-kube-api-access-lb2nz\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:13:19.784784 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:19.784635 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9ccc9b-9902-4c90-b9de-c06204a36322-maas-api-tls\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:13:20.376443 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:20.376365 2564 generic.go:356] "Generic (PLEG): container finished" podID="6e9ccc9b-9902-4c90-b9de-c06204a36322" containerID="ddde4535d5b481c26e6c4c17f816d6f64f8b20bd86917129c753d3da50ea57e2" exitCode=0 Jul 23 01:13:20.376443 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:20.376419 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" Jul 23 01:13:20.376934 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:20.376453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" event={"ID":"6e9ccc9b-9902-4c90-b9de-c06204a36322","Type":"ContainerDied","Data":"ddde4535d5b481c26e6c4c17f816d6f64f8b20bd86917129c753d3da50ea57e2"} Jul 23 01:13:20.376934 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:20.376504 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc" event={"ID":"6e9ccc9b-9902-4c90-b9de-c06204a36322","Type":"ContainerDied","Data":"55e3732f2ba8c04c51e050fb876d3520a5f095860b4fb3e036b3cff1c2bb6ebd"} Jul 23 01:13:20.376934 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:20.376548 2564 scope.go:117] "RemoveContainer" containerID="ddde4535d5b481c26e6c4c17f816d6f64f8b20bd86917129c753d3da50ea57e2" Jul 23 01:13:20.392991 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:20.392971 2564 scope.go:117] "RemoveContainer" containerID="ddde4535d5b481c26e6c4c17f816d6f64f8b20bd86917129c753d3da50ea57e2" Jul 23 01:13:20.393241 ip-10-0-139-1 kubenswrapper[2564]: E0723 01:13:20.393219 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddde4535d5b481c26e6c4c17f816d6f64f8b20bd86917129c753d3da50ea57e2\": container with ID starting with ddde4535d5b481c26e6c4c17f816d6f64f8b20bd86917129c753d3da50ea57e2 not found: ID does not exist" containerID="ddde4535d5b481c26e6c4c17f816d6f64f8b20bd86917129c753d3da50ea57e2" Jul 23 01:13:20.393286 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:20.393250 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddde4535d5b481c26e6c4c17f816d6f64f8b20bd86917129c753d3da50ea57e2"} err="failed to get container status \"ddde4535d5b481c26e6c4c17f816d6f64f8b20bd86917129c753d3da50ea57e2\": rpc error: code = NotFound desc = could not find container \"ddde4535d5b481c26e6c4c17f816d6f64f8b20bd86917129c753d3da50ea57e2\": container with ID starting with ddde4535d5b481c26e6c4c17f816d6f64f8b20bd86917129c753d3da50ea57e2 not found: ID does not exist" Jul 23 01:13:20.398042 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:20.398018 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc"] Jul 23 01:13:20.403312 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:20.403289 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-7ee5a5-7f64446df5-rxjbc"] Jul 23 01:13:22.072656 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:22.072627 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e9ccc9b-9902-4c90-b9de-c06204a36322" path="/var/lib/kubelet/pods/6e9ccc9b-9902-4c90-b9de-c06204a36322/volumes" Jul 23 01:13:23.747627 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:23.747592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:23.769274 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:23.769250 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j"] Jul 23 01:13:23.771548 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:23.771516 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-7ee5a5-nmj4j"] Jul 23 01:13:24.073435 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:24.073357 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0710c809-44a1-4f85-9419-c37a0b11f59e" path="/var/lib/kubelet/pods/0710c809-44a1-4f85-9419-c37a0b11f59e/volumes" Jul 23 01:13:24.158910 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:24.158880 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:28.363546 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:28.363418 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 01:13:28.398343 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:28.375150 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hq5hv_e4123100-2f7f-4b30-834f-04d2de11303b/ovn-acl-logging/0.log" Jul 23 01:13:28.477298 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:28.477269 2564 scope.go:117] "RemoveContainer" containerID="379f0a5a80aa603e903a2ce029ee48c742a98706183c64aae28a32785866903f" Jul 23 01:13:30.033725 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.033692 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:30.094013 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.093987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:30.316442 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.316356 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:30.548405 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.548371 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:30.681180 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.681156 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6"] Jul 23 01:13:30.681558 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.681545 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e9ccc9b-9902-4c90-b9de-c06204a36322" containerName="maas-api" Jul 23 01:13:30.681613 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.681559 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e9ccc9b-9902-4c90-b9de-c06204a36322" containerName="maas-api" Jul 23 01:13:30.681613 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.681571 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0710c809-44a1-4f85-9419-c37a0b11f59e" containerName="revoke-keys" Jul 23 01:13:30.681613 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.681579 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0710c809-44a1-4f85-9419-c37a0b11f59e" containerName="revoke-keys" Jul 23 01:13:30.681710 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.681652 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e9ccc9b-9902-4c90-b9de-c06204a36322" containerName="maas-api" Jul 23 01:13:30.681710 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.681660 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0710c809-44a1-4f85-9419-c37a0b11f59e" containerName="revoke-keys" Jul 23 01:13:30.686705 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.686688 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6" Jul 23 01:13:30.690380 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.690145 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zd6c\"" Jul 23 01:13:30.690380 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.690164 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 01:13:30.692830 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.692808 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6"] Jul 23 01:13:30.771628 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.771591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9685111e-dcb7-49ee-a418-cb937879127d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6\" (UID: \"9685111e-dcb7-49ee-a418-cb937879127d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6" Jul 23 01:13:30.872929 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.872886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9685111e-dcb7-49ee-a418-cb937879127d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6\" (UID: \"9685111e-dcb7-49ee-a418-cb937879127d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6" Jul 23 01:13:30.873473 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.873451 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9685111e-dcb7-49ee-a418-cb937879127d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6\" (UID: \"9685111e-dcb7-49ee-a418-cb937879127d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6" Jul 23 01:13:30.998859 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:30.998791 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6" Jul 23 01:13:31.131078 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:31.131048 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6"] Jul 23 01:13:31.132983 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:13:31.132959 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9685111e_dcb7_49ee_a418_cb937879127d.slice/crio-415151cf2b505f22be43f49a7c9a7d0374a064b04bcb3def4b308c80be308560 WatchSource:0}: Error finding container 415151cf2b505f22be43f49a7c9a7d0374a064b04bcb3def4b308c80be308560: Status 404 returned error can't find the container with id 415151cf2b505f22be43f49a7c9a7d0374a064b04bcb3def4b308c80be308560 Jul 23 01:13:31.312627 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:31.312599 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:31.414353 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:31.414320 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:31.443027 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:31.443003 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6" event={"ID":"9685111e-dcb7-49ee-a418-cb937879127d","Type":"ContainerStarted","Data":"3aea3487d96642bd4f841f716d5ae1bfbeb7ea58b795a33301f0b8e94a4ecee8"} Jul 23 01:13:31.443130 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:31.443078 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6" event={"ID":"9685111e-dcb7-49ee-a418-cb937879127d","Type":"ContainerStarted","Data":"415151cf2b505f22be43f49a7c9a7d0374a064b04bcb3def4b308c80be308560"} Jul 23 01:13:31.460805 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:31.460765 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6" podStartSLOduration=1.460750334 podStartE2EDuration="1.460750334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:13:31.457779554 +0000 UTC m=+3003.953448269" watchObservedRunningTime="2026-07-23 01:13:31.460750334 +0000 UTC m=+3003.956419048" Jul 23 01:13:32.449653 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:32.449619 2564 generic.go:356] "Generic (PLEG): container finished" podID="9685111e-dcb7-49ee-a418-cb937879127d" containerID="3aea3487d96642bd4f841f716d5ae1bfbeb7ea58b795a33301f0b8e94a4ecee8" exitCode=0 Jul 23 01:13:32.450139 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:32.449714 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6" event={"ID":"9685111e-dcb7-49ee-a418-cb937879127d","Type":"ContainerDied","Data":"3aea3487d96642bd4f841f716d5ae1bfbeb7ea58b795a33301f0b8e94a4ecee8"} Jul 23 01:13:33.588003 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:33.587978 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6" Jul 23 01:13:33.700926 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:33.700896 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9685111e-dcb7-49ee-a418-cb937879127d-maas-api-service-ca\") pod \"9685111e-dcb7-49ee-a418-cb937879127d\" (UID: \"9685111e-dcb7-49ee-a418-cb937879127d\") " Jul 23 01:13:33.701233 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:33.701211 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9685111e-dcb7-49ee-a418-cb937879127d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9685111e-dcb7-49ee-a418-cb937879127d" (UID: "9685111e-dcb7-49ee-a418-cb937879127d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 01:13:33.801667 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:33.801593 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9685111e-dcb7-49ee-a418-cb937879127d-maas-api-service-ca\") on node \"ip-10-0-139-1.ec2.internal\" DevicePath \"\"" Jul 23 01:13:34.462491 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:34.462460 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6" Jul 23 01:13:34.462675 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:34.462509 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6" event={"ID":"9685111e-dcb7-49ee-a418-cb937879127d","Type":"ContainerDied","Data":"415151cf2b505f22be43f49a7c9a7d0374a064b04bcb3def4b308c80be308560"} Jul 23 01:13:34.462675 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:34.462579 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="415151cf2b505f22be43f49a7c9a7d0374a064b04bcb3def4b308c80be308560" Jul 23 01:13:41.160772 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:41.160735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:45.759496 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:45.759461 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:45.759885 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:45.759840 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6"] Jul 23 01:13:45.764381 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:45.764357 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-d6869a-qvgj6"] Jul 23 01:13:46.073871 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:46.073779 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9685111e-dcb7-49ee-a418-cb937879127d" path="/var/lib/kubelet/pods/9685111e-dcb7-49ee-a418-cb937879127d/volumes" Jul 23 01:13:46.139878 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:46.139847 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:52.061929 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:52.061893 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:52.126204 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:52.126169 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:52.336219 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:52.336144 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:52.561060 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:52.561019 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:13:53.013499 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:53.013471 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68f94f797b-pghml_fb55a988-9291-44a6-b276-f26226fce713/authorino/0.log" Jul 23 01:13:59.218706 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:59.218678 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-695645577-b6l8r_546b7cfd-e56a-413e-b157-8a1810969aec/copy-manifests/0.log" Jul 23 01:13:59.225381 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:13:59.225357 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-695645577-b6l8r_546b7cfd-e56a-413e-b157-8a1810969aec/manager/0.log" Jul 23 01:14:00.109753 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:00.109721 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29746140-d7dq5_e5d3dadb-1ef5-4c10-915c-38152ce5d26c/cleanup/0.log" Jul 23 01:14:01.082886 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:01.082840 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz_02a19ed9-900d-4de6-89e6-ffcd229a4b9a/util/0.log" Jul 23 01:14:01.089318 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:01.089294 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz_02a19ed9-900d-4de6-89e6-ffcd229a4b9a/pull/0.log" Jul 23 01:14:01.094987 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:01.094962 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz_02a19ed9-900d-4de6-89e6-ffcd229a4b9a/extract/0.log" Jul 23 01:14:01.209867 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:01.209840 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm_e83cf781-638d-4728-bc84-7e3dc7fb8089/util/0.log" Jul 23 01:14:01.216003 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:01.215977 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm_e83cf781-638d-4728-bc84-7e3dc7fb8089/pull/0.log" Jul 23 01:14:01.221042 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:01.221025 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm_e83cf781-638d-4728-bc84-7e3dc7fb8089/extract/0.log" Jul 23 01:14:01.337926 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:01.337845 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv_b3c100f9-9818-4a56-80a3-77544ae18cf1/util/0.log" Jul 23 01:14:01.343452 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:01.343427 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv_b3c100f9-9818-4a56-80a3-77544ae18cf1/pull/0.log" Jul 23 01:14:01.349080 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:01.349054 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv_b3c100f9-9818-4a56-80a3-77544ae18cf1/extract/0.log" Jul 23 01:14:01.467762 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:01.467731 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r_87859bfc-58b8-4234-acc0-bbc1e5e1baa2/util/0.log" Jul 23 01:14:01.474326 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:01.474302 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r_87859bfc-58b8-4234-acc0-bbc1e5e1baa2/pull/0.log" Jul 23 01:14:01.480605 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:01.480586 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r_87859bfc-58b8-4234-acc0-bbc1e5e1baa2/extract/0.log" Jul 23 01:14:01.605716 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:01.605637 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68f94f797b-pghml_fb55a988-9291-44a6-b276-f26226fce713/authorino/0.log" Jul 23 01:14:02.872600 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:02.872558 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4pxsjg_080c6928-b5fe-4a13-bba0-4ae0ef99aa7f/istio-proxy/0.log" Jul 23 01:14:03.249327 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:03.249293 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-678df7485d-lxtmr_c2241c6c-9470-4742-9352-0418c07a566b/kube-auth-proxy/0.log" Jul 23 01:14:03.375551 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:03.375504 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-9srtx_d5b3b006-3ea6-445d-bf61-b1fed46711f0/istio-proxy/1.log" Jul 23 01:14:04.098698 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:04.098664 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-4ttm9_3fce81ce-1e9e-4430-915c-631e557f633f/main/0.log" Jul 23 01:14:04.900068 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:04.900034 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:14:04.954138 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:04.954099 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:14:11.115273 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.115238 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tpcmt/must-gather-dkskt"] Jul 23 01:14:11.115965 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.115939 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9685111e-dcb7-49ee-a418-cb937879127d" containerName="revoke-keys" Jul 23 01:14:11.116112 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.116037 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9685111e-dcb7-49ee-a418-cb937879127d" containerName="revoke-keys" Jul 23 01:14:11.116213 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.116189 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9685111e-dcb7-49ee-a418-cb937879127d" containerName="revoke-keys" Jul 23 01:14:11.119636 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.119617 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tpcmt/must-gather-dkskt" Jul 23 01:14:11.122264 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.122243 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tpcmt\"/\"default-dockercfg-j8gvw\"" Jul 23 01:14:11.122355 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.122246 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tpcmt\"/\"openshift-service-ca.crt\"" Jul 23 01:14:11.123394 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.123377 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tpcmt\"/\"kube-root-ca.crt\"" Jul 23 01:14:11.129471 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.129451 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tpcmt/must-gather-dkskt"] Jul 23 01:14:11.136743 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.136720 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:14:11.143751 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.143727 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dc97a462-a7b5-4512-b85f-7d815814389f-must-gather-output\") pod \"must-gather-dkskt\" (UID: \"dc97a462-a7b5-4512-b85f-7d815814389f\") " pod="openshift-must-gather-tpcmt/must-gather-dkskt" Jul 23 01:14:11.143859 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.143764 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhwzx\" (UniqueName: \"kubernetes.io/projected/dc97a462-a7b5-4512-b85f-7d815814389f-kube-api-access-jhwzx\") pod \"must-gather-dkskt\" (UID: \"dc97a462-a7b5-4512-b85f-7d815814389f\") " pod="openshift-must-gather-tpcmt/must-gather-dkskt" Jul 23 01:14:11.202604 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.202583 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" limit=1024 Jul 23 01:14:11.244341 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.244314 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dc97a462-a7b5-4512-b85f-7d815814389f-must-gather-output\") pod \"must-gather-dkskt\" (UID: \"dc97a462-a7b5-4512-b85f-7d815814389f\") " pod="openshift-must-gather-tpcmt/must-gather-dkskt" Jul 23 01:14:11.244439 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.244353 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhwzx\" (UniqueName: \"kubernetes.io/projected/dc97a462-a7b5-4512-b85f-7d815814389f-kube-api-access-jhwzx\") pod \"must-gather-dkskt\" (UID: \"dc97a462-a7b5-4512-b85f-7d815814389f\") " pod="openshift-must-gather-tpcmt/must-gather-dkskt" Jul 23 01:14:11.244670 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.244652 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dc97a462-a7b5-4512-b85f-7d815814389f-must-gather-output\") pod \"must-gather-dkskt\" (UID: \"dc97a462-a7b5-4512-b85f-7d815814389f\") " pod="openshift-must-gather-tpcmt/must-gather-dkskt" Jul 23 01:14:11.256005 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.255981 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhwzx\" (UniqueName: \"kubernetes.io/projected/dc97a462-a7b5-4512-b85f-7d815814389f-kube-api-access-jhwzx\") pod \"must-gather-dkskt\" (UID: \"dc97a462-a7b5-4512-b85f-7d815814389f\") " pod="openshift-must-gather-tpcmt/must-gather-dkskt" Jul 23 01:14:11.431149 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.431122 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tpcmt/must-gather-dkskt" Jul 23 01:14:11.574175 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.574139 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tpcmt/must-gather-dkskt"] Jul 23 01:14:11.575431 ip-10-0-139-1 kubenswrapper[2564]: W0723 01:14:11.575407 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc97a462_a7b5_4512_b85f_7d815814389f.slice/crio-47128aa814aa16456b65a8ed16ca9290d8a55f5d90180a54ffc95f7d2c01372d WatchSource:0}: Error finding container 47128aa814aa16456b65a8ed16ca9290d8a55f5d90180a54ffc95f7d2c01372d: Status 404 returned error can't find the container with id 47128aa814aa16456b65a8ed16ca9290d8a55f5d90180a54ffc95f7d2c01372d Jul 23 01:14:11.672303 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:11.672272 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpcmt/must-gather-dkskt" event={"ID":"dc97a462-a7b5-4512-b85f-7d815814389f","Type":"ContainerStarted","Data":"47128aa814aa16456b65a8ed16ca9290d8a55f5d90180a54ffc95f7d2c01372d"} Jul 23 01:14:12.686835 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:12.686568 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpcmt/must-gather-dkskt" event={"ID":"dc97a462-a7b5-4512-b85f-7d815814389f","Type":"ContainerStarted","Data":"c423c036eb56ae9c115245c4c28be3eeed08b26dbf16a02b7eafa1348498f55c"} Jul 23 01:14:12.686835 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:12.686621 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpcmt/must-gather-dkskt" event={"ID":"dc97a462-a7b5-4512-b85f-7d815814389f","Type":"ContainerStarted","Data":"89ad0d85e1c48cd30cf15d567d0e7271dc2e7ef8a8773c1550f8547179a54856"} Jul 23 01:14:12.703382 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:12.703321 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tpcmt/must-gather-dkskt" podStartSLOduration=0.925034325 podStartE2EDuration="1.703297118s" totalImagesPullingTime="778.262793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 01:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:14:12.7022965 +0000 UTC m=+3045.197965238" watchObservedRunningTime="2026-07-23 01:14:12.703297118 +0000 UTC m=+3045.198966037" Jul 23 01:14:14.000985 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:14.000953 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-njhfs_75dffbc4-21a5-4051-8cae-f7f9988f6630/global-pull-secret-syncer/0.log" Jul 23 01:14:14.050448 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:14.050405 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-82f2h_6411344d-2ba4-418f-9b2c-e5cd631325b6/konnectivity-agent/0.log" Jul 23 01:14:14.175179 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:14.175141 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-1.ec2.internal_4359bec0e352431350db386bdadf6398/haproxy/0.log" Jul 23 01:14:18.652812 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:18.652768 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz_02a19ed9-900d-4de6-89e6-ffcd229a4b9a/extract/0.log" Jul 23 01:14:18.675213 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:18.675147 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz_02a19ed9-900d-4de6-89e6-ffcd229a4b9a/util/0.log" Jul 23 01:14:18.697054 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:18.696983 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759pw5qz_02a19ed9-900d-4de6-89e6-ffcd229a4b9a/pull/0.log" Jul 23 01:14:18.737770 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:18.737742 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm_e83cf781-638d-4728-bc84-7e3dc7fb8089/extract/0.log" Jul 23 01:14:18.756818 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:18.756792 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm_e83cf781-638d-4728-bc84-7e3dc7fb8089/util/0.log" Jul 23 01:14:18.778553 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:18.778510 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e07s9jm_e83cf781-638d-4728-bc84-7e3dc7fb8089/pull/0.log" Jul 23 01:14:18.806289 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:18.806255 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv_b3c100f9-9818-4a56-80a3-77544ae18cf1/extract/0.log" Jul 23 01:14:18.826569 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:18.826543 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv_b3c100f9-9818-4a56-80a3-77544ae18cf1/util/0.log" Jul 23 01:14:18.846828 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:18.846785 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fnmkv_b3c100f9-9818-4a56-80a3-77544ae18cf1/pull/0.log" Jul 23 01:14:18.873993 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:18.873907 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r_87859bfc-58b8-4234-acc0-bbc1e5e1baa2/extract/0.log" Jul 23 01:14:18.903706 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:18.903592 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r_87859bfc-58b8-4234-acc0-bbc1e5e1baa2/util/0.log" Jul 23 01:14:18.927123 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:18.927095 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mwn8r_87859bfc-58b8-4234-acc0-bbc1e5e1baa2/pull/0.log" Jul 23 01:14:19.400351 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:19.400305 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68f94f797b-pghml_fb55a988-9291-44a6-b276-f26226fce713/authorino/0.log" Jul 23 01:14:21.461690 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:21.461659 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d38addb7-debd-46e7-b185-77d46cd891ec/alertmanager/0.log" Jul 23 01:14:21.490717 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:21.490684 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d38addb7-debd-46e7-b185-77d46cd891ec/config-reloader/0.log" Jul 23 01:14:21.518260 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:21.518181 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d38addb7-debd-46e7-b185-77d46cd891ec/kube-rbac-proxy-web/0.log" Jul 23 01:14:21.545432 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:21.545403 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d38addb7-debd-46e7-b185-77d46cd891ec/kube-rbac-proxy/0.log" Jul 23 01:14:21.572613 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:21.572583 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d38addb7-debd-46e7-b185-77d46cd891ec/kube-rbac-proxy-metric/0.log" Jul 23 01:14:21.596211 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:21.596177 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d38addb7-debd-46e7-b185-77d46cd891ec/prom-label-proxy/0.log" Jul 23 01:14:21.622309 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:21.622284 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d38addb7-debd-46e7-b185-77d46cd891ec/init-config-reloader/0.log" Jul 23 01:14:21.707869 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:21.707841 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-jm5s2_066d1d20-c2c9-4a0f-8979-bd68e53154b7/kube-state-metrics/0.log" Jul 23 01:14:21.730285 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:21.730260 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-jm5s2_066d1d20-c2c9-4a0f-8979-bd68e53154b7/kube-rbac-proxy-main/0.log" Jul 23 01:14:21.760772 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:21.760737 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-jm5s2_066d1d20-c2c9-4a0f-8979-bd68e53154b7/kube-rbac-proxy-self/0.log" Jul 23 01:14:21.796761 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:21.796665 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-75d4cd5994-995z5_1031cf51-37be-47f2-a48c-3c2bd09bb529/metrics-server/0.log" Jul 23 01:14:21.860482 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:21.860455 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r8pxs_0546d769-8f4f-4c68-be13-64bab60debb1/node-exporter/0.log" Jul 23 01:14:21.930471 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:21.930444 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r8pxs_0546d769-8f4f-4c68-be13-64bab60debb1/kube-rbac-proxy/0.log" Jul 23 01:14:21.979388 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:21.979361 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r8pxs_0546d769-8f4f-4c68-be13-64bab60debb1/init-textfile/0.log" Jul 23 01:14:22.189274 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:22.189245 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-cj7k6_ddac8ecd-70f8-498d-a8a9-eb9ff2278241/kube-rbac-proxy-main/0.log" Jul 23 01:14:22.211026 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:22.210959 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-cj7k6_ddac8ecd-70f8-498d-a8a9-eb9ff2278241/kube-rbac-proxy-self/0.log" Jul 23 01:14:22.231484 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:22.231441 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-cj7k6_ddac8ecd-70f8-498d-a8a9-eb9ff2278241/openshift-state-metrics/0.log" Jul 23 01:14:22.527659 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:22.527584 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d5b7988fb-5mbxj_ca25d7bb-3038-4662-9616-55e6dcb34f49/telemeter-client/0.log" Jul 23 01:14:22.548716 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:22.548685 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d5b7988fb-5mbxj_ca25d7bb-3038-4662-9616-55e6dcb34f49/reload/0.log" Jul 23 01:14:22.569150 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:22.569123 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d5b7988fb-5mbxj_ca25d7bb-3038-4662-9616-55e6dcb34f49/kube-rbac-proxy/0.log" Jul 23 01:14:22.601209 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:22.601184 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6575888f5c-dqsr5_630d7df4-b577-4d44-a4b5-204656fceb1e/thanos-query/0.log" Jul 23 01:14:22.621914 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:22.621886 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6575888f5c-dqsr5_630d7df4-b577-4d44-a4b5-204656fceb1e/kube-rbac-proxy-web/0.log" Jul 23 01:14:22.641939 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:22.641905 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6575888f5c-dqsr5_630d7df4-b577-4d44-a4b5-204656fceb1e/kube-rbac-proxy/0.log" Jul 23 01:14:22.662320 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:22.662289 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6575888f5c-dqsr5_630d7df4-b577-4d44-a4b5-204656fceb1e/prom-label-proxy/0.log" Jul 23 01:14:22.682354 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:22.682323 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6575888f5c-dqsr5_630d7df4-b577-4d44-a4b5-204656fceb1e/kube-rbac-proxy-rules/0.log" Jul 23 01:14:22.702951 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:22.702922 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6575888f5c-dqsr5_630d7df4-b577-4d44-a4b5-204656fceb1e/kube-rbac-proxy-metrics/0.log" Jul 23 01:14:23.821460 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.821424 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5"] Jul 23 01:14:23.829487 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.829457 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:23.834808 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.834779 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5"] Jul 23 01:14:23.890905 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.890563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/097b8047-03d2-4a62-91c0-250e12d7686c-lib-modules\") pod \"perf-node-gather-daemonset-2q7g5\" (UID: \"097b8047-03d2-4a62-91c0-250e12d7686c\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:23.890905 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.890703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/097b8047-03d2-4a62-91c0-250e12d7686c-proc\") pod \"perf-node-gather-daemonset-2q7g5\" (UID: \"097b8047-03d2-4a62-91c0-250e12d7686c\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:23.890905 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.890787 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/097b8047-03d2-4a62-91c0-250e12d7686c-podres\") pod \"perf-node-gather-daemonset-2q7g5\" (UID: \"097b8047-03d2-4a62-91c0-250e12d7686c\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:23.890905 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.890855 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/097b8047-03d2-4a62-91c0-250e12d7686c-sys\") pod \"perf-node-gather-daemonset-2q7g5\" (UID: \"097b8047-03d2-4a62-91c0-250e12d7686c\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:23.891234 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.890997 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zr2m\" (UniqueName: \"kubernetes.io/projected/097b8047-03d2-4a62-91c0-250e12d7686c-kube-api-access-7zr2m\") pod \"perf-node-gather-daemonset-2q7g5\" (UID: \"097b8047-03d2-4a62-91c0-250e12d7686c\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:23.992102 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.992070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/097b8047-03d2-4a62-91c0-250e12d7686c-podres\") pod \"perf-node-gather-daemonset-2q7g5\" (UID: \"097b8047-03d2-4a62-91c0-250e12d7686c\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:23.992355 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.992336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/097b8047-03d2-4a62-91c0-250e12d7686c-sys\") pod \"perf-node-gather-daemonset-2q7g5\" (UID: \"097b8047-03d2-4a62-91c0-250e12d7686c\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:23.992551 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.992254 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/097b8047-03d2-4a62-91c0-250e12d7686c-podres\") pod \"perf-node-gather-daemonset-2q7g5\" (UID: \"097b8047-03d2-4a62-91c0-250e12d7686c\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:23.992644 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.992416 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/097b8047-03d2-4a62-91c0-250e12d7686c-sys\") pod \"perf-node-gather-daemonset-2q7g5\" (UID: \"097b8047-03d2-4a62-91c0-250e12d7686c\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:23.992644 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.992511 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zr2m\" (UniqueName: \"kubernetes.io/projected/097b8047-03d2-4a62-91c0-250e12d7686c-kube-api-access-7zr2m\") pod \"perf-node-gather-daemonset-2q7g5\" (UID: \"097b8047-03d2-4a62-91c0-250e12d7686c\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:23.992757 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.992655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/097b8047-03d2-4a62-91c0-250e12d7686c-lib-modules\") pod \"perf-node-gather-daemonset-2q7g5\" (UID: \"097b8047-03d2-4a62-91c0-250e12d7686c\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:23.992823 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.992752 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/097b8047-03d2-4a62-91c0-250e12d7686c-proc\") pod \"perf-node-gather-daemonset-2q7g5\" (UID: \"097b8047-03d2-4a62-91c0-250e12d7686c\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:23.992883 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.992873 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/097b8047-03d2-4a62-91c0-250e12d7686c-proc\") pod \"perf-node-gather-daemonset-2q7g5\" (UID: \"097b8047-03d2-4a62-91c0-250e12d7686c\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:23.992938 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:23.992855 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/097b8047-03d2-4a62-91c0-250e12d7686c-lib-modules\") pod \"perf-node-gather-daemonset-2q7g5\" (UID: \"097b8047-03d2-4a62-91c0-250e12d7686c\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:24.002231 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:24.002209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zr2m\" (UniqueName: \"kubernetes.io/projected/097b8047-03d2-4a62-91c0-250e12d7686c-kube-api-access-7zr2m\") pod \"perf-node-gather-daemonset-2q7g5\" (UID: \"097b8047-03d2-4a62-91c0-250e12d7686c\") " pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:24.145217 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:24.145186 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:24.349960 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:24.349693 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5"] Jul 23 01:14:24.799112 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:24.799025 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" event={"ID":"097b8047-03d2-4a62-91c0-250e12d7686c","Type":"ContainerStarted","Data":"13dc603948c0cb30b195fe80847c7c32b4055cdc61465b8058fe5ea36e2aa33a"} Jul 23 01:14:24.799112 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:24.799087 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" event={"ID":"097b8047-03d2-4a62-91c0-250e12d7686c","Type":"ContainerStarted","Data":"edb70324797fe4008bf1b0673441372d01dd17fbb6640f07fcc91f4755d302d1"} Jul 23 01:14:24.799379 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:24.799249 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:24.816058 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:24.816012 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" podStartSLOduration=1.815997387 podStartE2EDuration="1.815997387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 01:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 01:14:24.8145992 +0000 UTC m=+3057.310267914" watchObservedRunningTime="2026-07-23 01:14:24.815997387 +0000 UTC m=+3057.311666100" Jul 23 01:14:25.037318 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:25.037283 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b756d7cd5-fdlsn_b2f00827-2629-4285-8b64-da272361f851/console/0.log" Jul 23 01:14:25.076749 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:25.076663 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-h2wtb_908cfd03-f8f2-424a-9802-76218e7d5128/download-server/0.log" Jul 23 01:14:25.631626 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:25.631600 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-h8gb6_5068b739-0dd4-4654-a415-417a309565be/volume-data-source-validator/0.log" Jul 23 01:14:26.658519 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:26.658487 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2gcnv_cba4baf3-9d99-4bb1-a822-8d80d49c1f9a/dns/0.log" Jul 23 01:14:26.682373 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:26.682348 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2gcnv_cba4baf3-9d99-4bb1-a822-8d80d49c1f9a/kube-rbac-proxy/0.log" Jul 23 01:14:26.852367 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:26.852329 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jl4hg_91698027-ce8c-4729-86e9-6eb62632951d/dns-node-resolver/0.log" Jul 23 01:14:27.475962 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:27.475930 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zxnfn_c0e82032-5917-427c-b7b6-e7597a1390d3/node-ca/0.log" Jul 23 01:14:28.392826 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:28.392793 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4pxsjg_080c6928-b5fe-4a13-bba0-4ae0ef99aa7f/istio-proxy/0.log" Jul 23 01:14:28.561659 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:28.561622 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-678df7485d-lxtmr_c2241c6c-9470-4742-9352-0418c07a566b/kube-auth-proxy/0.log" Jul 23 01:14:28.591678 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:28.591645 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-9srtx_d5b3b006-3ea6-445d-bf61-b1fed46711f0/istio-proxy/0.log" Jul 23 01:14:28.627185 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:28.627133 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-9srtx_d5b3b006-3ea6-445d-bf61-b1fed46711f0/istio-proxy/1.log" Jul 23 01:14:29.369604 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:29.369568 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kdnq4_e5461539-d692-499b-9b64-4d5b6def6be5/serve-healthcheck-canary/0.log" Jul 23 01:14:29.887351 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:29.887320 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-wjhdx_436060b9-c207-4a56-a70e-65e1a51c3b17/insights-operator/1.log" Jul 23 01:14:29.889391 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:29.889367 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-wjhdx_436060b9-c207-4a56-a70e-65e1a51c3b17/insights-operator/0.log" Jul 23 01:14:30.034501 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:30.034479 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p2xg4_cc4dfee5-f1cd-4494-90ca-c68bfef5f363/kube-rbac-proxy/0.log" Jul 23 01:14:30.054607 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:30.054577 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p2xg4_cc4dfee5-f1cd-4494-90ca-c68bfef5f363/exporter/0.log" Jul 23 01:14:30.083490 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:30.083457 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p2xg4_cc4dfee5-f1cd-4494-90ca-c68bfef5f363/extractor/0.log" Jul 23 01:14:30.820780 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:30.820749 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tpcmt/perf-node-gather-daemonset-2q7g5" Jul 23 01:14:32.649029 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:32.648994 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-695645577-b6l8r_546b7cfd-e56a-413e-b157-8a1810969aec/manager/0.log" Jul 23 01:14:32.674214 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:32.674181 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-695645577-b6l8r_546b7cfd-e56a-413e-b157-8a1810969aec/copy-manifests/0.log" Jul 23 01:14:34.240282 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:34.240200 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-85b95d447c-skrdm_77d9a36b-e5dd-478e-bdaa-aa4e1880b647/manager/0.log" Jul 23 01:14:34.423270 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:34.423199 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4pxsjg_080c6928-b5fe-4a13-bba0-4ae0ef99aa7f/istio-proxy/0.log" Jul 23 01:14:34.570840 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:34.570764 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-678df7485d-lxtmr_c2241c6c-9470-4742-9352-0418c07a566b/kube-auth-proxy/0.log" Jul 23 01:14:34.592335 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:34.592305 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-9srtx_d5b3b006-3ea6-445d-bf61-b1fed46711f0/istio-proxy/0.log" Jul 23 01:14:34.631834 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:34.631806 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-9srtx_d5b3b006-3ea6-445d-bf61-b1fed46711f0/istio-proxy/1.log" Jul 23 01:14:35.877877 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:35.877848 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4pxsjg_080c6928-b5fe-4a13-bba0-4ae0ef99aa7f/istio-proxy/0.log" Jul 23 01:14:36.034660 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:36.034580 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-678df7485d-lxtmr_c2241c6c-9470-4742-9352-0418c07a566b/kube-auth-proxy/0.log" Jul 23 01:14:36.067783 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:36.067747 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-9srtx_d5b3b006-3ea6-445d-bf61-b1fed46711f0/istio-proxy/0.log" Jul 23 01:14:36.105909 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:36.105878 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-9srtx_d5b3b006-3ea6-445d-bf61-b1fed46711f0/istio-proxy/1.log" Jul 23 01:14:39.215948 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:39.215914 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-cwk9s_5b4cf7c3-4507-4a93-a2ca-7a4ced835de5/migrator/0.log" Jul 23 01:14:39.236143 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:39.236121 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-cwk9s_5b4cf7c3-4507-4a93-a2ca-7a4ced835de5/graceful-termination/0.log" Jul 23 01:14:40.929741 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:40.929713 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h6c6l_c3e1769c-38a9-4493-8e79-c758ea29b40a/kube-multus-additional-cni-plugins/0.log" Jul 23 01:14:40.949165 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:40.949138 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h6c6l_c3e1769c-38a9-4493-8e79-c758ea29b40a/egress-router-binary-copy/0.log" Jul 23 01:14:40.969364 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:40.969337 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h6c6l_c3e1769c-38a9-4493-8e79-c758ea29b40a/cni-plugins/0.log" Jul 23 01:14:40.990569 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:40.990547 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h6c6l_c3e1769c-38a9-4493-8e79-c758ea29b40a/bond-cni-plugin/0.log" Jul 23 01:14:41.012049 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:41.012026 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h6c6l_c3e1769c-38a9-4493-8e79-c758ea29b40a/routeoverride-cni/0.log" Jul 23 01:14:41.031706 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:41.031689 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h6c6l_c3e1769c-38a9-4493-8e79-c758ea29b40a/whereabouts-cni-bincopy/0.log" Jul 23 01:14:41.050872 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:41.050852 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h6c6l_c3e1769c-38a9-4493-8e79-c758ea29b40a/whereabouts-cni/0.log" Jul 23 01:14:41.091364 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:41.091344 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-tf94d_581c6c94-dd79-49de-9cc9-fcbd6f40bcaf/kube-multus/0.log" Jul 23 01:14:41.192272 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:41.192185 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mx6kd_9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246/network-metrics-daemon/0.log" Jul 23 01:14:41.212445 ip-10-0-139-1 kubenswrapper[2564]: I0723 01:14:41.212422 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mx6kd_9a38f4c1-017f-4c0c-b9c2-4e60e9f2e246/kube-rbac-proxy/0.log"