Jul 05 19:28:16.881299 ip-10-0-138-40 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 05 19:28:16.881320 ip-10-0-138-40 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 05 19:28:16.881330 ip-10-0-138-40 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 05 19:28:16.881688 ip-10-0-138-40 systemd[1]: Failed to start Kubernetes Kubelet. Jul 05 19:28:26.893271 ip-10-0-138-40 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 05 19:28:26.893288 ip-10-0-138-40 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot be19a0d6665b4cf8bed20b9683db0676 -- Jul 05 19:29:57.175253 ip-10-0-138-40 systemd[1]: Starting Kubernetes Kubelet... Jul 05 19:29:57.613816 ip-10-0-138-40 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 05 19:29:57.613816 ip-10-0-138-40 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 05 19:29:57.613816 ip-10-0-138-40 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 05 19:29:57.613816 ip-10-0-138-40 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 05 19:29:57.613816 ip-10-0-138-40 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 05 19:29:57.614847 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.614737 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 05 19:29:57.620173 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620156 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 19:29:57.620173 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620174 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620178 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620182 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620184 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620187 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620190 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620192 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620195 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620197 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620200 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620202 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620205 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620207 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620210 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620213 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620215 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620218 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620222 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620227 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620230 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 19:29:57.620239 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620232 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620235 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620238 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620242 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620245 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620248 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620251 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620254 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620256 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620259 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620261 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620264 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620266 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620269 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620271 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620274 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620276 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620279 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620281 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 19:29:57.620721 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620283 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620286 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620290 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620294 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620297 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620299 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620302 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620304 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620306 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620309 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620311 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620313 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620316 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620318 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620320 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620323 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620326 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620329 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620331 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620334 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 19:29:57.621193 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620336 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620339 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620341 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620343 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620346 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620348 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620350 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620353 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620355 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620358 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620360 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620364 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620366 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620368 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620371 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620373 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620375 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620377 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620380 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620382 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 19:29:57.621684 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620385 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620387 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620389 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620392 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620394 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620397 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620400 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620403 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620406 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620409 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620412 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620414 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620417 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620419 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620421 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620424 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620426 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620428 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 19:29:57.622192 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.620431 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 19:29:57.622678 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622664 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 19:29:57.622678 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622678 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622682 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622684 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622687 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622691 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622693 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622696 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622699 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622701 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622704 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622706 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622709 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622711 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622714 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622716 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622719 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622721 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622724 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622727 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622729 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 19:29:57.622729 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622732 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622735 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622737 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622740 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622743 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622745 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622747 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622750 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622752 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622755 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622757 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622759 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622762 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622764 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622767 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622769 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622771 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622776 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622778 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622781 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 19:29:57.623240 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622783 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622787 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622791 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622793 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622796 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622799 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622802 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622805 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622808 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622810 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622813 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622815 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622818 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622820 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622823 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622826 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622828 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622831 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622834 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 19:29:57.623734 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622836 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622838 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622841 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622843 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622846 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622848 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622852 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622855 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622858 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622861 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622864 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622867 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622870 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622873 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622875 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622878 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622880 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622883 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622886 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622889 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 19:29:57.624215 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622892 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622895 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622897 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622899 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622902 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622905 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622907 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622923 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622926 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622930 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622934 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622937 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622940 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622943 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622945 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622948 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622950 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622953 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.622955 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 19:29:57.624726 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623322 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623328 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623332 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623335 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623339 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623341 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623344 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623363 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623367 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623370 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623372 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623376 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623378 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623381 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623384 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623386 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623389 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623391 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623394 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623396 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 19:29:57.625200 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623400 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623402 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623405 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623407 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623410 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623412 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623415 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623417 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623420 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623422 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623425 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623428 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623430 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623432 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623435 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623437 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623440 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623443 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623445 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 19:29:57.625780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623447 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623450 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623452 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623454 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623458 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623460 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623462 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623465 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623467 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623470 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623473 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623475 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623477 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623480 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623483 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623485 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623488 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623490 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623493 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 19:29:57.626307 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623495 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623499 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623502 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623505 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623507 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623511 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623515 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623519 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623521 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623524 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623527 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623530 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623533 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623535 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623539 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623541 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623544 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623547 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623550 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 19:29:57.626767 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623552 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623555 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623558 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623560 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623563 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623565 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623567 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623570 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623572 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623575 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623577 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623579 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623582 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623584 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623586 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623589 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623591 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623594 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623596 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623599 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 19:29:57.627256 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623601 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.623603 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623678 2573 flags.go:64] FLAG: --address="0.0.0.0" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623688 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623699 2573 flags.go:64] FLAG: --anonymous-auth="true" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623706 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623713 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623717 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623721 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623726 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623729 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623732 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623736 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623739 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623742 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623745 2573 flags.go:64] FLAG: --cgroup-root="" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623748 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623751 2573 flags.go:64] FLAG: --client-ca-file="" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623754 2573 flags.go:64] FLAG: --cloud-provider="external" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623757 2573 flags.go:64] FLAG: --cluster-dns="[]" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623761 2573 flags.go:64] FLAG: --cluster-domain="" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623763 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623766 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623769 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 05 19:29:57.627771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623772 2573 flags.go:64] FLAG: --container-log-max-files="5" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623776 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623779 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623782 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623785 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623788 2573 flags.go:64] FLAG: --contention-profiling="false" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623791 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623793 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623796 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623799 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623804 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623806 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623809 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623812 2573 flags.go:64] FLAG: --enable-load-reader="false" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623815 2573 flags.go:64] FLAG: --enable-server="true" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623818 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623823 2573 flags.go:64] FLAG: --event-burst="100" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623826 2573 flags.go:64] FLAG: --event-qps="50" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623830 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623833 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623837 2573 flags.go:64] FLAG: --eviction-hard="" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623841 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623843 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623846 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623849 2573 flags.go:64] FLAG: --eviction-soft="" Jul 05 19:29:57.628356 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623852 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623855 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623858 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623860 2573 flags.go:64] FLAG: --experimental-mounter-path="" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623863 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623866 2573 flags.go:64] FLAG: --fail-swap-on="true" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623868 2573 flags.go:64] FLAG: --feature-gates="" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623872 2573 flags.go:64] FLAG: --file-check-frequency="20s" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623875 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623878 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623881 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623884 2573 flags.go:64] FLAG: --healthz-port="10248" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623887 2573 flags.go:64] FLAG: --help="false" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623889 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-138-40.ec2.internal" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623892 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623895 2573 flags.go:64] FLAG: --http-check-frequency="20s" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623898 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623901 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623904 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623907 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623922 2573 flags.go:64] FLAG: --image-service-endpoint="" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623925 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623928 2573 flags.go:64] FLAG: --kube-api-burst="100" Jul 05 19:29:57.628968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623935 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623938 2573 flags.go:64] FLAG: --kube-api-qps="50" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623940 2573 flags.go:64] FLAG: --kube-reserved="" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623944 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623946 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623949 2573 flags.go:64] FLAG: --kubelet-cgroups="" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623952 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623955 2573 flags.go:64] FLAG: --lock-file="" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623957 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623960 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623963 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623968 2573 flags.go:64] FLAG: --log-json-split-stream="false" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623970 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623973 2573 flags.go:64] FLAG: --log-text-split-stream="false" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623976 2573 flags.go:64] FLAG: --logging-format="text" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623979 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623982 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623984 2573 flags.go:64] FLAG: --manifest-url="" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623987 2573 flags.go:64] FLAG: --manifest-url-header="" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623991 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623994 2573 flags.go:64] FLAG: --max-open-files="1000000" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.623998 2573 flags.go:64] FLAG: --max-pods="110" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624001 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624003 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624006 2573 flags.go:64] FLAG: --memory-manager-policy="None" Jul 05 19:29:57.629553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624008 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624011 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624014 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624016 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624023 2573 flags.go:64] FLAG: --node-status-max-images="50" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624026 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624029 2573 flags.go:64] FLAG: --oom-score-adj="-999" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624033 2573 flags.go:64] FLAG: --pod-cidr="" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624036 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624042 2573 flags.go:64] FLAG: --pod-manifest-path="" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624045 2573 flags.go:64] FLAG: --pod-max-pids="-1" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624047 2573 flags.go:64] FLAG: --pods-per-core="0" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624050 2573 flags.go:64] FLAG: --port="10250" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624053 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624056 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-012e41f478f311592" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624059 2573 flags.go:64] FLAG: --qos-reserved="" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624062 2573 flags.go:64] FLAG: --read-only-port="10255" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624064 2573 flags.go:64] FLAG: --register-node="true" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624067 2573 flags.go:64] FLAG: --register-with-taints="" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624070 2573 flags.go:64] FLAG: --registry-burst="10" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624073 2573 flags.go:64] FLAG: --registry-qps="5" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624076 2573 flags.go:64] FLAG: --reserved-cpus="" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624079 2573 flags.go:64] FLAG: --reserved-memory="" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624082 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624085 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 05 19:29:57.630151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624088 2573 flags.go:64] FLAG: --rotate-certificates="false" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624091 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624093 2573 flags.go:64] FLAG: --runonce="false" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624096 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624099 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624102 2573 flags.go:64] FLAG: --seccomp-default="false" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624105 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624107 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624110 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624113 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624115 2573 flags.go:64] FLAG: --storage-driver-password="root" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624118 2573 flags.go:64] FLAG: --storage-driver-secure="false" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624121 2573 flags.go:64] FLAG: --storage-driver-table="stats" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624123 2573 flags.go:64] FLAG: --storage-driver-user="root" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624127 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624130 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624133 2573 flags.go:64] FLAG: --system-cgroups="" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624136 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624141 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624144 2573 flags.go:64] FLAG: --tls-cert-file="" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624146 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624150 2573 flags.go:64] FLAG: --tls-min-version="" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624153 2573 flags.go:64] FLAG: --tls-private-key-file="" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624155 2573 flags.go:64] FLAG: --topology-manager-policy="none" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624158 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 05 19:29:57.630772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624160 2573 flags.go:64] FLAG: --topology-manager-scope="container" Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624163 2573 flags.go:64] FLAG: --v="2" Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624167 2573 flags.go:64] FLAG: --version="false" Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624171 2573 flags.go:64] FLAG: --vmodule="" Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624178 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.624181 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624299 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624302 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624306 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624310 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624313 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624315 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624318 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624320 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624323 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624325 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624328 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624330 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624333 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624335 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 19:29:57.631465 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624338 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624342 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624345 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624348 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624350 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624353 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624355 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624358 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624360 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624363 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624365 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624367 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624370 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624372 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624374 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624377 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624379 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624381 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624384 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 19:29:57.631988 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624386 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624390 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624393 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624396 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624399 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624402 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624404 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624407 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624410 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624412 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624415 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624418 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624421 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624424 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624428 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624432 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624435 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624437 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624440 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624442 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 19:29:57.632452 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624445 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624448 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624451 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624455 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624457 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624460 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624463 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624465 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624468 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624470 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624472 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624475 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624477 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624480 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624482 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624485 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624487 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624490 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624492 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624494 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 19:29:57.632952 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624497 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624499 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624502 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624504 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624506 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624509 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624512 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624516 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624519 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624521 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624524 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624527 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624529 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624532 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624534 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624536 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624539 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624541 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624544 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624546 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 19:29:57.633426 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624548 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 19:29:57.633924 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624551 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 19:29:57.633924 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624554 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 19:29:57.633924 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624556 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 19:29:57.633924 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624558 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 19:29:57.633924 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.624561 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 19:29:57.633924 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.625173 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 05 19:29:57.633924 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.632431 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 05 19:29:57.633924 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.632447 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 05 19:29:57.634248 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.634237 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 05 19:29:57.634280 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.634248 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 05 19:29:57.634319 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634312 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 19:29:57.634319 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634317 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634321 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634324 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634327 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634330 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634332 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634335 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634338 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634340 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634343 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634345 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634348 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634350 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634353 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634355 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634357 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634360 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634362 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634365 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634367 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634370 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 19:29:57.634372 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634372 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634374 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634377 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634380 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634383 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634386 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634388 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634392 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634395 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634399 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634403 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634406 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634408 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634412 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634414 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634417 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634419 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634422 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634425 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 19:29:57.634878 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634427 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634430 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634432 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634435 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634437 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634440 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634442 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634445 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634447 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634449 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634452 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634454 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634457 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634459 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634461 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634465 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634467 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634470 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634473 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634475 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 19:29:57.635434 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634478 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634481 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634483 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634486 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634488 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634491 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634493 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634495 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634498 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634500 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634502 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634505 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634507 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634510 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634513 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634517 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634519 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634522 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634525 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634527 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 19:29:57.635932 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634530 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634532 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634535 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634537 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634539 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634541 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634544 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634547 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634550 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634552 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634554 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634557 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634559 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634562 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634565 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634567 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634570 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 19:29:57.636453 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634572 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 19:29:57.636862 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.634577 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 05 19:29:57.636862 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634709 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 19:29:57.636862 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634714 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 19:29:57.636862 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634717 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 19:29:57.636862 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634720 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 19:29:57.636862 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634722 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 19:29:57.636862 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634725 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 19:29:57.636862 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634727 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 19:29:57.636862 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634729 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 19:29:57.636862 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634732 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 19:29:57.636862 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634734 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 19:29:57.636862 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634736 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 19:29:57.636862 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634739 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 19:29:57.636862 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634741 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 19:29:57.636862 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634745 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634749 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634752 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634754 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634758 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634761 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634765 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634768 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634771 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634774 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634777 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634779 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634781 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634784 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634786 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634789 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634792 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634794 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634797 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 19:29:57.637251 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634799 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634801 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634804 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634806 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634808 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634811 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634813 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634816 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634818 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634821 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634823 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634825 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634828 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634830 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634833 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634835 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634837 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634840 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634842 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 19:29:57.637758 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634844 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634847 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634850 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634853 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634855 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634857 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634860 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634862 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634865 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634867 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634870 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634873 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634876 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634878 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634880 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634883 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634885 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634888 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634890 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634893 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 19:29:57.638237 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634895 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634897 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634900 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634902 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634904 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634907 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634925 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634928 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634931 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634933 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634936 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634938 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634941 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634943 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634946 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634949 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634951 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634954 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634956 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634958 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 19:29:57.638738 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634961 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 19:29:57.639317 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634963 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 19:29:57.639317 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634966 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 19:29:57.639317 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634969 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 19:29:57.639317 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634971 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 19:29:57.639317 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634973 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 19:29:57.639317 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634976 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 19:29:57.639317 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:57.634978 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 19:29:57.639317 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.634982 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 05 19:29:57.639317 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.635098 2573 server.go:962] "Client rotation is on, will bootstrap in background" Jul 05 19:29:57.639317 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.637460 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 05 19:29:57.639317 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.638812 2573 server.go:1019] "Starting client certificate rotation" Jul 05 19:29:57.639317 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.638924 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 05 19:29:57.639317 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.638989 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 05 19:29:57.641945 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.641927 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 05 19:29:57.645712 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.645688 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 05 19:29:57.656776 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.656759 2573 log.go:25] "Validated CRI v1 runtime API" Jul 05 19:29:57.659436 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.659420 2573 log.go:25] "Validated CRI v1 image API" Jul 05 19:29:57.660198 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.660178 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 05 19:29:57.663495 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.663471 2573 fs.go:135] Filesystem UUIDs: map[5ed8dcf3-7a87-4f7b-8385-3c249a428205:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b34ae868-4929-4426-98b8-8a6d65c37cc6:/dev/nvme0n1p4] Jul 05 19:29:57.663549 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.663496 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 05 19:29:57.666959 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.666944 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 05 19:29:57.668974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.668846 2573 manager.go:216] Machine: {Timestamp:2026-07-05 19:29:57.668351115 +0000 UTC m=+0.361498133 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103065 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2edfd147991041c098e128a8be518d SystemUUID:ec2edfd1-4799-1041-c098-e128a8be518d BootID:be19a0d6-665b-4cf8-bed2-0b9683db0676 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:97:72:ed:b0:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:97:72:ed:b0:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:62:4c:62:cc:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 05 19:29:57.668974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.668970 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 05 19:29:57.669064 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.669049 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 05 19:29:57.670304 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.670270 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 05 19:29:57.670446 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.670307 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-40.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 05 19:29:57.671566 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.671556 2573 topology_manager.go:138] "Creating topology manager with none policy" Jul 05 19:29:57.671597 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.671569 2573 container_manager_linux.go:309] "Creating device plugin manager" Jul 05 19:29:57.671597 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.671577 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 05 19:29:57.672247 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.672237 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 05 19:29:57.673676 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.673666 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 05 19:29:57.678302 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.678282 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 05 19:29:57.678421 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.678411 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 05 19:29:57.678456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.678442 2573 kubelet.go:486] "Attempting to sync node with API server" Jul 05 19:29:57.678456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.678452 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 05 19:29:57.678516 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.678466 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 05 19:29:57.678516 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.678477 2573 kubelet.go:398] "Adding apiserver pod source" Jul 05 19:29:57.678516 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.678486 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 05 19:29:57.679006 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.678961 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 05 19:29:57.679006 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.678980 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 05 19:29:57.683964 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.683950 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 05 19:29:57.686374 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.686361 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 05 19:29:57.686424 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.686391 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 05 19:29:57.687773 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.687761 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 05 19:29:57.687810 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.687779 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 05 19:29:57.687810 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.687790 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 05 19:29:57.687810 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.687796 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 05 19:29:57.687810 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.687805 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 05 19:29:57.687810 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.687810 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 05 19:29:57.687996 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.687819 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 05 19:29:57.687996 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.687825 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 05 19:29:57.687996 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.687834 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 05 19:29:57.687996 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.687842 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 05 19:29:57.687996 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.687862 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 05 19:29:57.687996 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.687872 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 05 19:29:57.688464 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.688441 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 05 19:29:57.688464 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.688452 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 05 19:29:57.689735 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.689699 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-40.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 05 19:29:57.689806 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.689712 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 05 19:29:57.691683 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.691671 2573 server.go:1268] "Started kubelet" Jul 05 19:29:57.691809 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.691750 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 05 19:29:57.695436 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.695411 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 05 19:29:57.695575 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.695549 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 05 19:29:57.695904 ip-10-0-138-40 systemd[1]: Started Kubernetes Kubelet. Jul 05 19:29:57.697898 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.697237 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 05 19:29:57.697898 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.697666 2573 server.go:310] "Adding debug handlers to kubelet server" Jul 05 19:29:57.701042 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.701025 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 05 19:29:57.701877 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.701864 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-40.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 05 19:29:57.702705 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.701890 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-40.ec2.internal.18bf7b871c8a73e9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-40.ec2.internal,UID:ip-10-0-138-40.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-40.ec2.internal,},FirstTimestamp:2026-07-05 19:29:57.691642857 +0000 UTC m=+0.384789874,LastTimestamp:2026-07-05 19:29:57.691642857 +0000 UTC m=+0.384789874,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-40.ec2.internal,}" Jul 05 19:29:57.705054 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.705039 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 05 19:29:57.705097 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.705054 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 05 19:29:57.705187 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.705166 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-40.ec2.internal\" not found" Jul 05 19:29:57.705783 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.705766 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 05 19:29:57.705870 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.705784 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 05 19:29:57.705870 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.705801 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 05 19:29:57.705986 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.705872 2573 factory.go:55] Registering systemd factory Jul 05 19:29:57.705986 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.705885 2573 factory.go:223] Registration of the systemd container factory successfully Jul 05 19:29:57.705986 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.705939 2573 reconstruct.go:97] "Volume reconstruction finished" Jul 05 19:29:57.705986 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.705948 2573 reconciler.go:29] "Reconciler: start to sync state" Jul 05 19:29:57.706393 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.706377 2573 factory.go:153] Registering CRI-O factory Jul 05 19:29:57.706393 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.706395 2573 factory.go:223] Registration of the crio container factory successfully Jul 05 19:29:57.706502 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.706441 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 05 19:29:57.706502 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.706468 2573 factory.go:103] Registering Raw factory Jul 05 19:29:57.706502 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.706482 2573 manager.go:1196] Started watching for new ooms in manager Jul 05 19:29:57.707175 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.707161 2573 manager.go:349] Starting recovery of all containers Jul 05 19:29:57.709726 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.709696 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 05 19:29:57.709827 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.709791 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-40.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 05 19:29:57.714741 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.714714 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rq7pj" Jul 05 19:29:57.716560 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.716545 2573 manager.go:354] Recovery completed Jul 05 19:29:57.719738 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.719623 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rq7pj" Jul 05 19:29:57.721991 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.721978 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 19:29:57.723800 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.723786 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasSufficientMemory" Jul 05 19:29:57.723885 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.723815 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 19:29:57.723885 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.723825 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasSufficientPID" Jul 05 19:29:57.724418 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.724404 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 19:29:57.724502 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.724406 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 05 19:29:57.724502 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.724434 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 05 19:29:57.724502 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.724449 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 05 19:29:57.725114 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.725100 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasSufficientMemory" Jul 05 19:29:57.725172 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.725123 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 19:29:57.725172 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.725132 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasSufficientPID" Jul 05 19:29:57.725422 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.725365 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-40.ec2.internal.18bf7b871e75228d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-40.ec2.internal,UID:ip-10-0-138-40.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-40.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-40.ec2.internal,},FirstTimestamp:2026-07-05 19:29:57.723800205 +0000 UTC m=+0.416947225,LastTimestamp:2026-07-05 19:29:57.723800205 +0000 UTC m=+0.416947225,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-40.ec2.internal,}" Jul 05 19:29:57.726430 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.726418 2573 policy_none.go:49] "None policy: Start" Jul 05 19:29:57.726474 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.726439 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 05 19:29:57.726474 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.726450 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 05 19:29:57.728348 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.728337 2573 policy_none.go:47] "Start" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.767980 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.769287 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.769335 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.769364 2573 kubelet.go:2448] "Starting kubelet main sync loop" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.769402 2573 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.771599 2573 manager.go:337] "Starting Device Plugin manager" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.771605 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.771636 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.771651 2573 server.go:87] "Starting device plugin registration server" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.771975 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.771990 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.772135 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.772218 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.772226 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.772669 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 05 19:29:57.783030 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.772717 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-40.ec2.internal\" not found" Jul 05 19:29:57.869806 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.869705 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-40.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal"] Jul 05 19:29:57.869958 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.869835 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 19:29:57.871746 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.871728 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasSufficientMemory" Jul 05 19:29:57.871843 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.871762 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 19:29:57.871843 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.871773 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasSufficientPID" Jul 05 19:29:57.872802 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.872790 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 19:29:57.873516 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.873497 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasSufficientMemory" Jul 05 19:29:57.873626 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.873522 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 19:29:57.873626 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.873539 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasSufficientPID" Jul 05 19:29:57.873626 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.873564 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-40.ec2.internal" Jul 05 19:29:57.874176 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.874164 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 19:29:57.874292 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.874257 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-40.ec2.internal" Jul 05 19:29:57.874331 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.874296 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 19:29:57.874772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.874757 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasSufficientMemory" Jul 05 19:29:57.874848 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.874779 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 19:29:57.874848 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.874791 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasSufficientPID" Jul 05 19:29:57.874935 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.874791 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasSufficientMemory" Jul 05 19:29:57.874935 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.874880 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 19:29:57.874935 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.874898 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasSufficientPID" Jul 05 19:29:57.877073 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.877061 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal" Jul 05 19:29:57.877179 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.877100 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 19:29:57.877753 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.877738 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasSufficientMemory" Jul 05 19:29:57.877812 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.877766 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 19:29:57.877812 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.877775 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeHasSufficientPID" Jul 05 19:29:57.881861 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.881841 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-40.ec2.internal" Jul 05 19:29:57.881954 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.881865 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-40.ec2.internal\": node \"ip-10-0-138-40.ec2.internal\" not found" Jul 05 19:29:57.893469 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.893450 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-40.ec2.internal\" not found" node="ip-10-0-138-40.ec2.internal" Jul 05 19:29:57.896686 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.896669 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-40.ec2.internal\" not found" node="ip-10-0-138-40.ec2.internal" Jul 05 19:29:57.905845 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:57.905830 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-40.ec2.internal\" not found" Jul 05 19:29:57.906859 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.906841 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c667c56910299e2ae0fff31b3d85acc6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-40.ec2.internal\" (UID: \"c667c56910299e2ae0fff31b3d85acc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-40.ec2.internal" Jul 05 19:29:57.906902 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.906867 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b9cf5d1d2b26f6e5b718a5e66d212ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal\" (UID: \"1b9cf5d1d2b26f6e5b718a5e66d212ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal" Jul 05 19:29:57.906902 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:57.906885 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b9cf5d1d2b26f6e5b718a5e66d212ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal\" (UID: \"1b9cf5d1d2b26f6e5b718a5e66d212ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal" Jul 05 19:29:58.005918 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:58.005894 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-40.ec2.internal\" not found" Jul 05 19:29:58.006999 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.006980 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c667c56910299e2ae0fff31b3d85acc6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-40.ec2.internal\" (UID: \"c667c56910299e2ae0fff31b3d85acc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-40.ec2.internal" Jul 05 19:29:58.007049 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.007008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b9cf5d1d2b26f6e5b718a5e66d212ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal\" (UID: \"1b9cf5d1d2b26f6e5b718a5e66d212ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal" Jul 05 19:29:58.007049 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.007025 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b9cf5d1d2b26f6e5b718a5e66d212ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal\" (UID: \"1b9cf5d1d2b26f6e5b718a5e66d212ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal" Jul 05 19:29:58.007111 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.007074 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b9cf5d1d2b26f6e5b718a5e66d212ac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal\" (UID: \"1b9cf5d1d2b26f6e5b718a5e66d212ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal" Jul 05 19:29:58.007111 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.007081 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c667c56910299e2ae0fff31b3d85acc6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-40.ec2.internal\" (UID: \"c667c56910299e2ae0fff31b3d85acc6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-40.ec2.internal" Jul 05 19:29:58.007111 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.007090 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1b9cf5d1d2b26f6e5b718a5e66d212ac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal\" (UID: \"1b9cf5d1d2b26f6e5b718a5e66d212ac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal" Jul 05 19:29:58.106190 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:58.106144 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-40.ec2.internal\" not found" Jul 05 19:29:58.196831 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.196756 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-40.ec2.internal" Jul 05 19:29:58.199892 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.199872 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal" Jul 05 19:29:58.206329 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:58.206310 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-40.ec2.internal\" not found" Jul 05 19:29:58.306783 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:58.306743 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-40.ec2.internal\" not found" Jul 05 19:29:58.407149 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:58.407111 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-40.ec2.internal\" not found" Jul 05 19:29:58.502368 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.502273 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 19:29:58.507854 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:58.507832 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-40.ec2.internal\" not found" Jul 05 19:29:58.608423 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:29:58.608382 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-40.ec2.internal\" not found" Jul 05 19:29:58.638646 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.638616 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 05 19:29:58.639208 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.638761 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 05 19:29:58.639208 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.638796 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 05 19:29:58.691876 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.691845 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 19:29:58.705744 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.705722 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-40.ec2.internal" Jul 05 19:29:58.705834 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.705722 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 05 19:29:58.716060 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.716037 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 05 19:29:58.717184 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.717165 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal" Jul 05 19:29:58.719823 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.719806 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 05 19:29:58.726069 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.726042 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-04 19:24:59 +0000 UTC" deadline="2028-03-05 14:04:37.578023315 +0000 UTC" Jul 05 19:29:58.726069 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.726065 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14610h34m38.8519613s" Jul 05 19:29:58.728740 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.728723 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 05 19:29:58.744393 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.744369 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rn7lr" Jul 05 19:29:58.752665 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.752618 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rn7lr" Jul 05 19:29:58.918075 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:58.918027 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc667c56910299e2ae0fff31b3d85acc6.slice/crio-3a519a27b28b221ba85f8e0434706374595f4cb420dbdc1e7caa6e3e24ad6199 WatchSource:0}: Error finding container 3a519a27b28b221ba85f8e0434706374595f4cb420dbdc1e7caa6e3e24ad6199: Status 404 returned error can't find the container with id 3a519a27b28b221ba85f8e0434706374595f4cb420dbdc1e7caa6e3e24ad6199 Jul 05 19:29:58.919300 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:29:58.919280 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9cf5d1d2b26f6e5b718a5e66d212ac.slice/crio-258e4e3f555cab18511d493a6d9f2f8debf095bbaaf8270e317f612956b5ba19 WatchSource:0}: Error finding container 258e4e3f555cab18511d493a6d9f2f8debf095bbaaf8270e317f612956b5ba19: Status 404 returned error can't find the container with id 258e4e3f555cab18511d493a6d9f2f8debf095bbaaf8270e317f612956b5ba19 Jul 05 19:29:58.923869 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.923856 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 05 19:29:58.997285 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:29:58.997254 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 19:30:01.372300 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.372212 2573 apiserver.go:52] "Watching apiserver" Jul 05 19:30:01.380894 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.380866 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 05 19:30:01.381402 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.381369 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw","openshift-image-registry/node-ca-4snds","openshift-multus/multus-additional-cni-plugins-v428c","openshift-network-diagnostics/network-check-target-qsk8d","openshift-network-operator/iptables-alerter-jx5q9","openshift-ovn-kubernetes/ovnkube-node-s4jh7","kube-system/global-pull-secret-syncer-pcsv4","openshift-cluster-node-tuning-operator/tuned-xvdjm","openshift-dns/node-resolver-xnmxp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal","openshift-multus/multus-l7gg4","openshift-multus/network-metrics-daemon-86bxp","kube-system/konnectivity-agent-ktlfv","kube-system/kube-apiserver-proxy-ip-10-0-138-40.ec2.internal"] Jul 05 19:30:01.383901 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.383878 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:01.384017 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:01.383994 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:01.386125 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.386105 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ktlfv" Jul 05 19:30:01.388369 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.388352 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.388574 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.388553 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 05 19:30:01.388788 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.388771 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-92clt\"" Jul 05 19:30:01.388882 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.388773 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 05 19:30:01.390512 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.390492 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:01.390613 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:01.390560 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:01.390945 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.390924 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 05 19:30:01.391687 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.391666 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 05 19:30:01.391808 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.391788 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pjjsq\"" Jul 05 19:30:01.391908 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.391851 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 05 19:30:01.391908 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.391859 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 05 19:30:01.392134 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.392120 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 05 19:30:01.393319 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.393302 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jx5q9" Jul 05 19:30:01.395578 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.395559 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 05 19:30:01.395934 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.395913 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 05 19:30:01.396118 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.396102 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nw77c\"" Jul 05 19:30:01.396190 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.396155 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.396716 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.396686 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 05 19:30:01.397820 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.397802 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xnmxp" Jul 05 19:30:01.399775 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.399753 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 05 19:30:01.399940 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.399922 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 05 19:30:01.399991 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.399945 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 05 19:30:01.400090 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.400013 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 05 19:30:01.400090 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.400022 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 05 19:30:01.400371 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.400343 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.400903 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.400885 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 05 19:30:01.400962 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.400917 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5lcm9\"" Jul 05 19:30:01.400962 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.400950 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 05 19:30:01.401026 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.400993 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 05 19:30:01.401026 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.400908 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v7q4h\"" Jul 05 19:30:01.404552 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.404533 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 05 19:30:01.405726 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.405707 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 05 19:30:01.405990 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.405974 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-j9z7w\"" Jul 05 19:30:01.406213 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.406197 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 05 19:30:01.406755 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.406738 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4snds" Jul 05 19:30:01.408631 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.408602 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjks5\" (UniqueName: \"kubernetes.io/projected/2b2a331e-da1e-4db5-ac53-84d90da88ff3-kube-api-access-zjks5\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.408732 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.408648 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-run-netns\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.408732 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.408683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-var-lib-openvswitch\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.408732 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.408724 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-ovn-node-metrics-cert\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.408845 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.408756 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-socket-dir\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.408845 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.408782 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-registration-dir\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.408845 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.408813 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b2a331e-da1e-4db5-ac53-84d90da88ff3-system-cni-dir\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.408932 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.408842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b2a331e-da1e-4db5-ac53-84d90da88ff3-cni-binary-copy\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.408932 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.408889 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-slash\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.408932 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.408926 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-etc-openvswitch\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.409035 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.408960 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-log-socket\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.409035 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.408995 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c3262-1cc8-4aea-a843-26a51eb356f8-konnectivity-ca\") pod \"konnectivity-agent-ktlfv\" (UID: \"dc4c3262-1cc8-4aea-a843-26a51eb356f8\") " pod="kube-system/konnectivity-agent-ktlfv" Jul 05 19:30:01.409132 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409036 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b2a331e-da1e-4db5-ac53-84d90da88ff3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.409132 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409095 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-cni-bin\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.409226 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409134 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2zw2\" (UniqueName: \"kubernetes.io/projected/5c96e505-de6a-4853-822c-e80de519e002-kube-api-access-v2zw2\") pod \"node-resolver-xnmxp\" (UID: \"5c96e505-de6a-4853-822c-e80de519e002\") " pod="openshift-dns/node-resolver-xnmxp" Jul 05 19:30:01.409226 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409165 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-device-dir\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.409226 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409200 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.409369 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409236 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7lmk\" (UniqueName: \"kubernetes.io/projected/9536cff9-3d7f-415b-aa0a-8a71113511d8-kube-api-access-x7lmk\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.409369 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409277 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b2a331e-da1e-4db5-ac53-84d90da88ff3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.409369 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409315 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-systemd-units\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.409369 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409342 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-run-openvswitch\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.409548 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409373 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-run-ovn\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.409548 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409417 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-cni-netd\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.409548 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409345 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.409663 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409576 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-sys-fs\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.409663 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409618 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d84a2165-5809-40c0-a5b3-7d3a976291ee-kubelet-config\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:01.409730 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409671 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:01.409730 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409697 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b2a331e-da1e-4db5-ac53-84d90da88ff3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.409730 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409726 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9d9c89b-7f73-4ca0-a000-1d01ab3ec580-host-slash\") pod \"iptables-alerter-jx5q9\" (UID: \"a9d9c89b-7f73-4ca0-a000-1d01ab3ec580\") " pod="openshift-network-operator/iptables-alerter-jx5q9" Jul 05 19:30:01.409824 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409756 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-kubelet\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.409824 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-run-systemd\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.409920 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409844 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.409920 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409872 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b2a331e-da1e-4db5-ac53-84d90da88ff3-cnibin\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.409920 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409902 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-node-log\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.410082 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409931 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-run-ovn-kubernetes\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.410082 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409961 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-ovnkube-script-lib\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.410082 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.409991 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkdbx\" (UniqueName: \"kubernetes.io/projected/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-kube-api-access-pkdbx\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.410082 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.410014 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c96e505-de6a-4853-822c-e80de519e002-tmp-dir\") pod \"node-resolver-xnmxp\" (UID: \"5c96e505-de6a-4853-822c-e80de519e002\") " pod="openshift-dns/node-resolver-xnmxp" Jul 05 19:30:01.410082 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.410064 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc4c3262-1cc8-4aea-a843-26a51eb356f8-agent-certs\") pod \"konnectivity-agent-ktlfv\" (UID: \"dc4c3262-1cc8-4aea-a843-26a51eb356f8\") " pod="kube-system/konnectivity-agent-ktlfv" Jul 05 19:30:01.410750 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.410092 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d84a2165-5809-40c0-a5b3-7d3a976291ee-dbus\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:01.410750 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.410181 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ptdc\" (UniqueName: \"kubernetes.io/projected/a9d9c89b-7f73-4ca0-a000-1d01ab3ec580-kube-api-access-6ptdc\") pod \"iptables-alerter-jx5q9\" (UID: \"a9d9c89b-7f73-4ca0-a000-1d01ab3ec580\") " pod="openshift-network-operator/iptables-alerter-jx5q9" Jul 05 19:30:01.410750 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.410232 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-ovnkube-config\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.410750 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.410280 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c96e505-de6a-4853-822c-e80de519e002-hosts-file\") pod \"node-resolver-xnmxp\" (UID: \"5c96e505-de6a-4853-822c-e80de519e002\") " pod="openshift-dns/node-resolver-xnmxp" Jul 05 19:30:01.410750 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.410309 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4j8b\" (UniqueName: \"kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b\") pod \"network-check-target-qsk8d\" (UID: \"de14a84c-5bb0-4f31-9894-c71098e82aec\") " pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:01.410750 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.410407 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.410750 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.410421 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a9d9c89b-7f73-4ca0-a000-1d01ab3ec580-iptables-alerter-script\") pod \"iptables-alerter-jx5q9\" (UID: \"a9d9c89b-7f73-4ca0-a000-1d01ab3ec580\") " pod="openshift-network-operator/iptables-alerter-jx5q9" Jul 05 19:30:01.410750 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.410463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.410750 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.410504 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-env-overrides\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.410750 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.410540 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2b2a331e-da1e-4db5-ac53-84d90da88ff3-os-release\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.411187 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.411170 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 05 19:30:01.411781 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.411760 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 05 19:30:01.411857 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.411839 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q8qvg\"" Jul 05 19:30:01.411953 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.411927 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:01.412059 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:01.412021 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:01.412157 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.412141 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 05 19:30:01.412256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.412182 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hnlqg\"" Jul 05 19:30:01.412342 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.411763 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 05 19:30:01.412407 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.412393 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 05 19:30:01.415638 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.415087 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 05 19:30:01.415638 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.415136 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jg89k\"" Jul 05 19:30:01.424143 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.424125 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 19:30:01.449109 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.449055 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-04 19:25:00 +0000 UTC" deadline="2028-04-21 00:25:58.763580418 +0000 UTC" Jul 05 19:30:01.449109 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.449109 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15724h55m57.314474906s" Jul 05 19:30:01.467962 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.467919 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-40.ec2.internal" event={"ID":"c667c56910299e2ae0fff31b3d85acc6","Type":"ContainerStarted","Data":"3a519a27b28b221ba85f8e0434706374595f4cb420dbdc1e7caa6e3e24ad6199"} Jul 05 19:30:01.469516 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.469490 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal" event={"ID":"1b9cf5d1d2b26f6e5b718a5e66d212ac","Type":"ContainerStarted","Data":"258e4e3f555cab18511d493a6d9f2f8debf095bbaaf8270e317f612956b5ba19"} Jul 05 19:30:01.500285 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.500202 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 05 19:30:01.510990 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.510956 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfaa8adb-2377-43ab-8481-f3d30e66a994-cni-binary-copy\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.510990 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.510994 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b2a331e-da1e-4db5-ac53-84d90da88ff3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.511222 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511022 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-systemd-units\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.511222 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511060 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-run-openvswitch\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.511222 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511087 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-run-ovn\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.511222 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511120 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-cni-netd\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.511222 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-sys-fs\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.511222 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d84a2165-5809-40c0-a5b3-7d3a976291ee-kubelet-config\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:01.511222 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511196 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9d9c89b-7f73-4ca0-a000-1d01ab3ec580-host-slash\") pod \"iptables-alerter-jx5q9\" (UID: \"a9d9c89b-7f73-4ca0-a000-1d01ab3ec580\") " pod="openshift-network-operator/iptables-alerter-jx5q9" Jul 05 19:30:01.511222 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c96e505-de6a-4853-822c-e80de519e002-tmp-dir\") pod \"node-resolver-xnmxp\" (UID: \"5c96e505-de6a-4853-822c-e80de519e002\") " pod="openshift-dns/node-resolver-xnmxp" Jul 05 19:30:01.511506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511231 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-sysctl-d\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.511506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511248 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-cnibin\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.511506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511281 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-run-netns\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.511506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511313 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b2a331e-da1e-4db5-ac53-84d90da88ff3-cnibin\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.511506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-node-log\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.511506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-ovnkube-script-lib\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.511506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511369 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkdbx\" (UniqueName: \"kubernetes.io/projected/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-kube-api-access-pkdbx\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.511506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511390 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-sysconfig\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.511506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511407 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-kubernetes\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.511506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511439 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-var-lib-cni-bin\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.511506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs\") pod \"network-metrics-daemon-86bxp\" (UID: \"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3\") " pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:01.511506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511495 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc4c3262-1cc8-4aea-a843-26a51eb356f8-agent-certs\") pod \"konnectivity-agent-ktlfv\" (UID: \"dc4c3262-1cc8-4aea-a843-26a51eb356f8\") " pod="kube-system/konnectivity-agent-ktlfv" Jul 05 19:30:01.511974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511523 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ptdc\" (UniqueName: \"kubernetes.io/projected/a9d9c89b-7f73-4ca0-a000-1d01ab3ec580-kube-api-access-6ptdc\") pod \"iptables-alerter-jx5q9\" (UID: \"a9d9c89b-7f73-4ca0-a000-1d01ab3ec580\") " pod="openshift-network-operator/iptables-alerter-jx5q9" Jul 05 19:30:01.511974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511546 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c96e505-de6a-4853-822c-e80de519e002-hosts-file\") pod \"node-resolver-xnmxp\" (UID: \"5c96e505-de6a-4853-822c-e80de519e002\") " pod="openshift-dns/node-resolver-xnmxp" Jul 05 19:30:01.511974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511578 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-systemd\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.511974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511600 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-tuned\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.511974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511621 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-hostroot\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.511974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511690 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8b\" (UniqueName: \"kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b\") pod \"network-check-target-qsk8d\" (UID: \"de14a84c-5bb0-4f31-9894-c71098e82aec\") " pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:01.511974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511722 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-sys\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.511974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511746 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-multus-conf-dir\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.511974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511775 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2b2a331e-da1e-4db5-ac53-84d90da88ff3-os-release\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.511974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511799 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjks5\" (UniqueName: \"kubernetes.io/projected/2b2a331e-da1e-4db5-ac53-84d90da88ff3-kube-api-access-zjks5\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.511974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511827 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-run-netns\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.511974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-var-lib-openvswitch\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.511974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511889 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-socket-dir\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.511974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511913 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b2a331e-da1e-4db5-ac53-84d90da88ff3-system-cni-dir\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.511974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511935 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-slash\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.511987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-cni-bin\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76pml\" (UniqueName: \"kubernetes.io/projected/cb2e6493-927f-48b2-8e47-16dc2379911a-kube-api-access-76pml\") pod \"node-ca-4snds\" (UID: \"cb2e6493-927f-48b2-8e47-16dc2379911a\") " pod="openshift-image-registry/node-ca-4snds" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512060 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfaa8adb-2377-43ab-8481-f3d30e66a994-multus-daemon-config\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512083 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-run-multus-certs\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512119 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-device-dir\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512202 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7lmk\" (UniqueName: \"kubernetes.io/projected/9536cff9-3d7f-415b-aa0a-8a71113511d8-kube-api-access-x7lmk\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512226 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-ovnkube-config\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512249 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-sysctl-conf\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512275 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-run\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb2e6493-927f-48b2-8e47-16dc2379911a-host\") pod \"node-ca-4snds\" (UID: \"cb2e6493-927f-48b2-8e47-16dc2379911a\") " pod="openshift-image-registry/node-ca-4snds" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512331 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb2e6493-927f-48b2-8e47-16dc2379911a-serviceca\") pod \"node-ca-4snds\" (UID: \"cb2e6493-927f-48b2-8e47-16dc2379911a\") " pod="openshift-image-registry/node-ca-4snds" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b2a331e-da1e-4db5-ac53-84d90da88ff3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512419 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-kubelet\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.512456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-run-systemd\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512473 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-run-ovn-kubernetes\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512491 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcr5v\" (UniqueName: \"kubernetes.io/projected/1380926c-7f7f-426e-8b55-b3015f6c9ad2-kube-api-access-bcr5v\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512505 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-run-k8s-cni-cncf-io\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512528 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512550 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-lib-modules\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512564 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-multus-socket-dir-parent\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512581 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-var-lib-kubelet\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d84a2165-5809-40c0-a5b3-7d3a976291ee-dbus\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512614 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1380926c-7f7f-426e-8b55-b3015f6c9ad2-tmp\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512628 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-var-lib-cni-multus\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512642 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjgxg\" (UniqueName: \"kubernetes.io/projected/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-kube-api-access-hjgxg\") pod \"network-metrics-daemon-86bxp\" (UID: \"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3\") " pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512657 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a9d9c89b-7f73-4ca0-a000-1d01ab3ec580-iptables-alerter-script\") pod \"iptables-alerter-jx5q9\" (UID: \"a9d9c89b-7f73-4ca0-a000-1d01ab3ec580\") " pod="openshift-network-operator/iptables-alerter-jx5q9" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512692 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512725 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-env-overrides\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512748 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-system-cni-dir\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512792 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-etc-kubernetes\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.512971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-ovn-node-metrics-cert\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512865 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-modprobe-d\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512895 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-registration-dir\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512922 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b2a331e-da1e-4db5-ac53-84d90da88ff3-cni-binary-copy\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-etc-openvswitch\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.512978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-log-socket\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513004 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-os-release\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513027 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g68qb\" (UniqueName: \"kubernetes.io/projected/cfaa8adb-2377-43ab-8481-f3d30e66a994-kube-api-access-g68qb\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513073 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c3262-1cc8-4aea-a843-26a51eb356f8-konnectivity-ca\") pod \"konnectivity-agent-ktlfv\" (UID: \"dc4c3262-1cc8-4aea-a843-26a51eb356f8\") " pod="kube-system/konnectivity-agent-ktlfv" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513100 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b2a331e-da1e-4db5-ac53-84d90da88ff3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2zw2\" (UniqueName: \"kubernetes.io/projected/5c96e505-de6a-4853-822c-e80de519e002-kube-api-access-v2zw2\") pod \"node-resolver-xnmxp\" (UID: \"5c96e505-de6a-4853-822c-e80de519e002\") " pod="openshift-dns/node-resolver-xnmxp" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513157 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-var-lib-kubelet\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513187 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-host\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513216 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-multus-cni-dir\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513459 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b2a331e-da1e-4db5-ac53-84d90da88ff3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513511 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-systemd-units\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513547 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-run-openvswitch\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.513612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513595 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-run-ovn\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.514338 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513633 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-cni-netd\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.514338 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513684 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-sys-fs\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.514338 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513745 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d84a2165-5809-40c0-a5b3-7d3a976291ee-kubelet-config\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:01.514338 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.513784 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9d9c89b-7f73-4ca0-a000-1d01ab3ec580-host-slash\") pod \"iptables-alerter-jx5q9\" (UID: \"a9d9c89b-7f73-4ca0-a000-1d01ab3ec580\") " pod="openshift-network-operator/iptables-alerter-jx5q9" Jul 05 19:30:01.514338 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.514078 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c96e505-de6a-4853-822c-e80de519e002-tmp-dir\") pod \"node-resolver-xnmxp\" (UID: \"5c96e505-de6a-4853-822c-e80de519e002\") " pod="openshift-dns/node-resolver-xnmxp" Jul 05 19:30:01.514338 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.514189 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b2a331e-da1e-4db5-ac53-84d90da88ff3-cnibin\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.514338 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.514229 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-node-log\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.514835 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.514816 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-ovnkube-script-lib\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.514994 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.514968 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-etc-openvswitch\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.515109 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.515011 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-kubelet\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.515109 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:01.515020 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:01.515109 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.515051 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-run-systemd\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.515109 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.515066 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-var-lib-openvswitch\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.515109 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.515085 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-run-ovn-kubernetes\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.515347 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:01.515136 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret podName:d84a2165-5809-40c0-a5b3-7d3a976291ee nodeName:}" failed. No retries permitted until 2026-07-05 19:30:02.01509663 +0000 UTC m=+3.015114729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret") pod "global-pull-secret-syncer-pcsv4" (UID: "d84a2165-5809-40c0-a5b3-7d3a976291ee") : object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:01.515347 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.515151 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.515347 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.515216 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b2a331e-da1e-4db5-ac53-84d90da88ff3-system-cni-dir\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.515347 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.515290 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-env-overrides\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.515347 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.515335 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c96e505-de6a-4853-822c-e80de519e002-hosts-file\") pod \"node-resolver-xnmxp\" (UID: \"5c96e505-de6a-4853-822c-e80de519e002\") " pod="openshift-dns/node-resolver-xnmxp" Jul 05 19:30:01.515593 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.515339 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-log-socket\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.515593 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.515372 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-socket-dir\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.515593 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.515528 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 05 19:30:01.515718 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.515592 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b2a331e-da1e-4db5-ac53-84d90da88ff3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.515802 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.515739 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-registration-dir\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.515857 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.515836 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2b2a331e-da1e-4db5-ac53-84d90da88ff3-os-release\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.515909 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.515884 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.516364 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.516300 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d84a2165-5809-40c0-a5b3-7d3a976291ee-dbus\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:01.516364 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.516346 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-cni-bin\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.516364 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.516346 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-ovnkube-config\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.516656 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.516407 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b2a331e-da1e-4db5-ac53-84d90da88ff3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.516656 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.516416 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-device-dir\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.516656 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.516303 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a9d9c89b-7f73-4ca0-a000-1d01ab3ec580-iptables-alerter-script\") pod \"iptables-alerter-jx5q9\" (UID: \"a9d9c89b-7f73-4ca0-a000-1d01ab3ec580\") " pod="openshift-network-operator/iptables-alerter-jx5q9" Jul 05 19:30:01.516656 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.516501 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-slash\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.516947 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.516915 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b2a331e-da1e-4db5-ac53-84d90da88ff3-cni-binary-copy\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.517136 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.516511 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9536cff9-3d7f-415b-aa0a-8a71113511d8-etc-selinux\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.517136 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.517001 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dc4c3262-1cc8-4aea-a843-26a51eb356f8-konnectivity-ca\") pod \"konnectivity-agent-ktlfv\" (UID: \"dc4c3262-1cc8-4aea-a843-26a51eb356f8\") " pod="kube-system/konnectivity-agent-ktlfv" Jul 05 19:30:01.517136 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.517066 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-host-run-netns\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.519935 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.519902 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-ovn-node-metrics-cert\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.521035 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.521018 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dc4c3262-1cc8-4aea-a843-26a51eb356f8-agent-certs\") pod \"konnectivity-agent-ktlfv\" (UID: \"dc4c3262-1cc8-4aea-a843-26a51eb356f8\") " pod="kube-system/konnectivity-agent-ktlfv" Jul 05 19:30:01.523168 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:01.523139 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 19:30:01.523168 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:01.523161 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 19:30:01.523338 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:01.523174 2573 projected.go:196] Error preparing data for projected volume kube-api-access-h4j8b for pod openshift-network-diagnostics/network-check-target-qsk8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:01.523338 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:01.523232 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b podName:de14a84c-5bb0-4f31-9894-c71098e82aec nodeName:}" failed. No retries permitted until 2026-07-05 19:30:02.023210709 +0000 UTC m=+3.023228798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h4j8b" (UniqueName: "kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b") pod "network-check-target-qsk8d" (UID: "de14a84c-5bb0-4f31-9894-c71098e82aec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:01.527080 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.527033 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjks5\" (UniqueName: \"kubernetes.io/projected/2b2a331e-da1e-4db5-ac53-84d90da88ff3-kube-api-access-zjks5\") pod \"multus-additional-cni-plugins-v428c\" (UID: \"2b2a331e-da1e-4db5-ac53-84d90da88ff3\") " pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.529178 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.529152 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkdbx\" (UniqueName: \"kubernetes.io/projected/46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c-kube-api-access-pkdbx\") pod \"ovnkube-node-s4jh7\" (UID: \"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c\") " pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.530326 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.530289 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ptdc\" (UniqueName: \"kubernetes.io/projected/a9d9c89b-7f73-4ca0-a000-1d01ab3ec580-kube-api-access-6ptdc\") pod \"iptables-alerter-jx5q9\" (UID: \"a9d9c89b-7f73-4ca0-a000-1d01ab3ec580\") " pod="openshift-network-operator/iptables-alerter-jx5q9" Jul 05 19:30:01.531457 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.530965 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2zw2\" (UniqueName: \"kubernetes.io/projected/5c96e505-de6a-4853-822c-e80de519e002-kube-api-access-v2zw2\") pod \"node-resolver-xnmxp\" (UID: \"5c96e505-de6a-4853-822c-e80de519e002\") " pod="openshift-dns/node-resolver-xnmxp" Jul 05 19:30:01.531457 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.531394 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7lmk\" (UniqueName: \"kubernetes.io/projected/9536cff9-3d7f-415b-aa0a-8a71113511d8-kube-api-access-x7lmk\") pod \"aws-ebs-csi-driver-node-smlfw\" (UID: \"9536cff9-3d7f-415b-aa0a-8a71113511d8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.614128 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614082 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-os-release\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.614128 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g68qb\" (UniqueName: \"kubernetes.io/projected/cfaa8adb-2377-43ab-8481-f3d30e66a994-kube-api-access-g68qb\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.614362 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614164 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-var-lib-kubelet\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.614362 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614182 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-host\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.614362 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614206 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-multus-cni-dir\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.614362 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614228 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfaa8adb-2377-43ab-8481-f3d30e66a994-cni-binary-copy\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.614362 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614257 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-sysctl-d\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.614362 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614278 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-cnibin\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.614362 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614288 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-var-lib-kubelet\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.614362 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614294 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-os-release\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.614362 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614311 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-host\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.614362 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614301 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-run-netns\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.614362 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614332 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-run-netns\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.614362 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614365 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-sysconfig\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614379 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-multus-cni-dir\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614390 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-kubernetes\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614418 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-cnibin\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614414 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-var-lib-cni-bin\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614448 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs\") pod \"network-metrics-daemon-86bxp\" (UID: \"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3\") " pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614450 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-var-lib-cni-bin\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614489 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-kubernetes\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614499 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-sysconfig\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614501 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-sysctl-d\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:01.614581 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614609 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-systemd\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:01.614639 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs podName:ed74973b-94ab-43a3-a0d3-b5e89daf5cf3 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:02.114617983 +0000 UTC m=+3.114636092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs") pod "network-metrics-daemon-86bxp" (UID: "ed74973b-94ab-43a3-a0d3-b5e89daf5cf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614660 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-systemd\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614664 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-tuned\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614688 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-hostroot\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614723 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-sys\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614745 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-multus-conf-dir\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.614922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614752 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-hostroot\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614778 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76pml\" (UniqueName: \"kubernetes.io/projected/cb2e6493-927f-48b2-8e47-16dc2379911a-kube-api-access-76pml\") pod \"node-ca-4snds\" (UID: \"cb2e6493-927f-48b2-8e47-16dc2379911a\") " pod="openshift-image-registry/node-ca-4snds" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614802 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-sys\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfaa8adb-2377-43ab-8481-f3d30e66a994-multus-daemon-config\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614829 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-run-multus-certs\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614840 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-multus-conf-dir\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614877 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-run-multus-certs\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614889 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cfaa8adb-2377-43ab-8481-f3d30e66a994-cni-binary-copy\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614907 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-sysctl-conf\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614931 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-run\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614951 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb2e6493-927f-48b2-8e47-16dc2379911a-host\") pod \"node-ca-4snds\" (UID: \"cb2e6493-927f-48b2-8e47-16dc2379911a\") " pod="openshift-image-registry/node-ca-4snds" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.614975 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb2e6493-927f-48b2-8e47-16dc2379911a-serviceca\") pod \"node-ca-4snds\" (UID: \"cb2e6493-927f-48b2-8e47-16dc2379911a\") " pod="openshift-image-registry/node-ca-4snds" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615018 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcr5v\" (UniqueName: \"kubernetes.io/projected/1380926c-7f7f-426e-8b55-b3015f6c9ad2-kube-api-access-bcr5v\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615028 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-sysctl-conf\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615032 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-run\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615065 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-run-k8s-cni-cncf-io\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615094 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-lib-modules\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615118 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-multus-socket-dir-parent\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.615633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615068 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb2e6493-927f-48b2-8e47-16dc2379911a-host\") pod \"node-ca-4snds\" (UID: \"cb2e6493-927f-48b2-8e47-16dc2379911a\") " pod="openshift-image-registry/node-ca-4snds" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615142 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-var-lib-kubelet\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615167 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1380926c-7f7f-426e-8b55-b3015f6c9ad2-tmp\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615174 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-run-k8s-cni-cncf-io\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615190 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-var-lib-cni-multus\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615213 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjgxg\" (UniqueName: \"kubernetes.io/projected/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-kube-api-access-hjgxg\") pod \"network-metrics-daemon-86bxp\" (UID: \"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3\") " pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615218 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-var-lib-kubelet\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615240 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-system-cni-dir\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615262 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-lib-modules\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615269 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-multus-socket-dir-parent\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615263 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-etc-kubernetes\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615307 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-etc-kubernetes\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615313 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cfaa8adb-2377-43ab-8481-f3d30e66a994-multus-daemon-config\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615347 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-modprobe-d\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615388 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-host-var-lib-cni-multus\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615440 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-modprobe-d\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615452 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cfaa8adb-2377-43ab-8481-f3d30e66a994-system-cni-dir\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.616256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.615516 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cb2e6493-927f-48b2-8e47-16dc2379911a-serviceca\") pod \"node-ca-4snds\" (UID: \"cb2e6493-927f-48b2-8e47-16dc2379911a\") " pod="openshift-image-registry/node-ca-4snds" Jul 05 19:30:01.617710 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.617688 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1380926c-7f7f-426e-8b55-b3015f6c9ad2-tmp\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.617820 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.617777 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1380926c-7f7f-426e-8b55-b3015f6c9ad2-etc-tuned\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.626666 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.626610 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcr5v\" (UniqueName: \"kubernetes.io/projected/1380926c-7f7f-426e-8b55-b3015f6c9ad2-kube-api-access-bcr5v\") pod \"tuned-xvdjm\" (UID: \"1380926c-7f7f-426e-8b55-b3015f6c9ad2\") " pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.626796 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.626774 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjgxg\" (UniqueName: \"kubernetes.io/projected/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-kube-api-access-hjgxg\") pod \"network-metrics-daemon-86bxp\" (UID: \"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3\") " pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:01.627236 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.627215 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g68qb\" (UniqueName: \"kubernetes.io/projected/cfaa8adb-2377-43ab-8481-f3d30e66a994-kube-api-access-g68qb\") pod \"multus-l7gg4\" (UID: \"cfaa8adb-2377-43ab-8481-f3d30e66a994\") " pod="openshift-multus/multus-l7gg4" Jul 05 19:30:01.628033 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.628014 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76pml\" (UniqueName: \"kubernetes.io/projected/cb2e6493-927f-48b2-8e47-16dc2379911a-kube-api-access-76pml\") pod \"node-ca-4snds\" (UID: \"cb2e6493-927f-48b2-8e47-16dc2379911a\") " pod="openshift-image-registry/node-ca-4snds" Jul 05 19:30:01.699111 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.699077 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ktlfv" Jul 05 19:30:01.708792 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.708767 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v428c" Jul 05 19:30:01.711394 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:30:01.711366 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4c3262_1cc8_4aea_a843_26a51eb356f8.slice/crio-121189d5eb2799b40a15ebe472bc6c067b9be8ce25deafd09d4b1db7d5e6357e WatchSource:0}: Error finding container 121189d5eb2799b40a15ebe472bc6c067b9be8ce25deafd09d4b1db7d5e6357e: Status 404 returned error can't find the container with id 121189d5eb2799b40a15ebe472bc6c067b9be8ce25deafd09d4b1db7d5e6357e Jul 05 19:30:01.720145 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:30:01.720103 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2a331e_da1e_4db5_ac53_84d90da88ff3.slice/crio-ada080f822f27ad9b8516ad48af02a9f79cde0e2ee5e78f1072cb501f2ce05b1 WatchSource:0}: Error finding container ada080f822f27ad9b8516ad48af02a9f79cde0e2ee5e78f1072cb501f2ce05b1: Status 404 returned error can't find the container with id ada080f822f27ad9b8516ad48af02a9f79cde0e2ee5e78f1072cb501f2ce05b1 Jul 05 19:30:01.722894 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.722870 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jx5q9" Jul 05 19:30:01.728832 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.728807 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:01.737901 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.737874 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xnmxp" Jul 05 19:30:01.745817 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.745796 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" Jul 05 19:30:01.755671 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.755652 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4snds" Jul 05 19:30:01.764543 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.764525 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" Jul 05 19:30:01.772363 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:01.772338 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l7gg4" Jul 05 19:30:02.017946 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:02.017852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:02.018142 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:02.018026 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:02.018142 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:02.018112 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret podName:d84a2165-5809-40c0-a5b3-7d3a976291ee nodeName:}" failed. No retries permitted until 2026-07-05 19:30:03.018090504 +0000 UTC m=+4.018108593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret") pod "global-pull-secret-syncer-pcsv4" (UID: "d84a2165-5809-40c0-a5b3-7d3a976291ee") : object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:02.118788 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:02.118739 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs\") pod \"network-metrics-daemon-86bxp\" (UID: \"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3\") " pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:02.118788 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:02.118793 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8b\" (UniqueName: \"kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b\") pod \"network-check-target-qsk8d\" (UID: \"de14a84c-5bb0-4f31-9894-c71098e82aec\") " pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:02.119006 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:02.118895 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:02.119006 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:02.118926 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 19:30:02.119006 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:02.118952 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 19:30:02.119006 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:02.118965 2573 projected.go:196] Error preparing data for projected volume kube-api-access-h4j8b for pod openshift-network-diagnostics/network-check-target-qsk8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:02.119006 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:02.118966 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs podName:ed74973b-94ab-43a3-a0d3-b5e89daf5cf3 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:03.118947224 +0000 UTC m=+4.118965388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs") pod "network-metrics-daemon-86bxp" (UID: "ed74973b-94ab-43a3-a0d3-b5e89daf5cf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:02.119249 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:02.119016 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b podName:de14a84c-5bb0-4f31-9894-c71098e82aec nodeName:}" failed. No retries permitted until 2026-07-05 19:30:03.119003141 +0000 UTC m=+4.119021224 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4j8b" (UniqueName: "kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b") pod "network-check-target-qsk8d" (UID: "de14a84c-5bb0-4f31-9894-c71098e82aec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:02.299052 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:02.298948 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 19:30:02.393144 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:30:02.393111 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d9c89b_7f73_4ca0_a000_1d01ab3ec580.slice/crio-3461705e31a3334cb078cc88860202cf3491bd4615789d19e4f1bca97fb156d7 WatchSource:0}: Error finding container 3461705e31a3334cb078cc88860202cf3491bd4615789d19e4f1bca97fb156d7: Status 404 returned error can't find the container with id 3461705e31a3334cb078cc88860202cf3491bd4615789d19e4f1bca97fb156d7 Jul 05 19:30:02.449527 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:02.449452 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-04 19:25:00 +0000 UTC" deadline="2028-03-01 03:35:00.284369617 +0000 UTC" Jul 05 19:30:02.449527 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:02.449522 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14504h4m57.83485022s" Jul 05 19:30:02.472130 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:02.472084 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jx5q9" event={"ID":"a9d9c89b-7f73-4ca0-a000-1d01ab3ec580","Type":"ContainerStarted","Data":"3461705e31a3334cb078cc88860202cf3491bd4615789d19e4f1bca97fb156d7"} Jul 05 19:30:02.473146 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:02.473109 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v428c" event={"ID":"2b2a331e-da1e-4db5-ac53-84d90da88ff3","Type":"ContainerStarted","Data":"ada080f822f27ad9b8516ad48af02a9f79cde0e2ee5e78f1072cb501f2ce05b1"} Jul 05 19:30:02.474192 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:02.474170 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ktlfv" event={"ID":"dc4c3262-1cc8-4aea-a843-26a51eb356f8","Type":"ContainerStarted","Data":"121189d5eb2799b40a15ebe472bc6c067b9be8ce25deafd09d4b1db7d5e6357e"} Jul 05 19:30:02.851817 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:30:02.851783 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46dae9aa_7ed9_4e1c_bb2c_f25dfd1b114c.slice/crio-762468b3eb00b3ff8578b1ae987b7e288f4f107233f0598fe9e58a439d882a0e WatchSource:0}: Error finding container 762468b3eb00b3ff8578b1ae987b7e288f4f107233f0598fe9e58a439d882a0e: Status 404 returned error can't find the container with id 762468b3eb00b3ff8578b1ae987b7e288f4f107233f0598fe9e58a439d882a0e Jul 05 19:30:02.853625 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:30:02.853587 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfaa8adb_2377_43ab_8481_f3d30e66a994.slice/crio-115f60156bafd54eadd746af39f3b7d19a33d2c3504a34d5d17e20f371413389 WatchSource:0}: Error finding container 115f60156bafd54eadd746af39f3b7d19a33d2c3504a34d5d17e20f371413389: Status 404 returned error can't find the container with id 115f60156bafd54eadd746af39f3b7d19a33d2c3504a34d5d17e20f371413389 Jul 05 19:30:02.879903 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:30:02.879878 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c96e505_de6a_4853_822c_e80de519e002.slice/crio-8897d37ecfaa968350d5eea450d439f0184d6778ef937586a894ef98ad185753 WatchSource:0}: Error finding container 8897d37ecfaa968350d5eea450d439f0184d6778ef937586a894ef98ad185753: Status 404 returned error can't find the container with id 8897d37ecfaa968350d5eea450d439f0184d6778ef937586a894ef98ad185753 Jul 05 19:30:02.881916 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:30:02.881878 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9536cff9_3d7f_415b_aa0a_8a71113511d8.slice/crio-3bb7c99c7eac0dcaf7e362dbe88cda819bd5f374f31df8c8d40482e197bde572 WatchSource:0}: Error finding container 3bb7c99c7eac0dcaf7e362dbe88cda819bd5f374f31df8c8d40482e197bde572: Status 404 returned error can't find the container with id 3bb7c99c7eac0dcaf7e362dbe88cda819bd5f374f31df8c8d40482e197bde572 Jul 05 19:30:03.025734 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:03.025374 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:03.025734 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:03.025675 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:03.025949 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:03.025757 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret podName:d84a2165-5809-40c0-a5b3-7d3a976291ee nodeName:}" failed. No retries permitted until 2026-07-05 19:30:05.025737484 +0000 UTC m=+6.025755571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret") pod "global-pull-secret-syncer-pcsv4" (UID: "d84a2165-5809-40c0-a5b3-7d3a976291ee") : object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:03.126507 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:03.126414 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs\") pod \"network-metrics-daemon-86bxp\" (UID: \"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3\") " pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:03.126507 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:03.126470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8b\" (UniqueName: \"kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b\") pod \"network-check-target-qsk8d\" (UID: \"de14a84c-5bb0-4f31-9894-c71098e82aec\") " pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:03.126707 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:03.126571 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:03.126707 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:03.126643 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs podName:ed74973b-94ab-43a3-a0d3-b5e89daf5cf3 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:05.126623637 +0000 UTC m=+6.126641721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs") pod "network-metrics-daemon-86bxp" (UID: "ed74973b-94ab-43a3-a0d3-b5e89daf5cf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:03.126707 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:03.126661 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 19:30:03.126707 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:03.126680 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 19:30:03.126707 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:03.126697 2573 projected.go:196] Error preparing data for projected volume kube-api-access-h4j8b for pod openshift-network-diagnostics/network-check-target-qsk8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:03.126947 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:03.126741 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b podName:de14a84c-5bb0-4f31-9894-c71098e82aec nodeName:}" failed. No retries permitted until 2026-07-05 19:30:05.126725601 +0000 UTC m=+6.126743685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4j8b" (UniqueName: "kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b") pod "network-check-target-qsk8d" (UID: "de14a84c-5bb0-4f31-9894-c71098e82aec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:03.463322 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:03.463257 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:03.463687 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:03.463378 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:03.463821 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:03.463798 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:03.463919 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:03.463897 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:03.464085 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:03.464066 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:03.464172 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:03.464150 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:03.476890 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:03.476859 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4snds" event={"ID":"cb2e6493-927f-48b2-8e47-16dc2379911a","Type":"ContainerStarted","Data":"49326e1da681039511725ad21c688e00deedc8486846f81029cb91250f17b191"} Jul 05 19:30:03.478303 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:03.478275 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" event={"ID":"9536cff9-3d7f-415b-aa0a-8a71113511d8","Type":"ContainerStarted","Data":"3bb7c99c7eac0dcaf7e362dbe88cda819bd5f374f31df8c8d40482e197bde572"} Jul 05 19:30:03.479890 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:03.479612 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l7gg4" event={"ID":"cfaa8adb-2377-43ab-8481-f3d30e66a994","Type":"ContainerStarted","Data":"115f60156bafd54eadd746af39f3b7d19a33d2c3504a34d5d17e20f371413389"} Jul 05 19:30:03.480927 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:03.480901 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" event={"ID":"1380926c-7f7f-426e-8b55-b3015f6c9ad2","Type":"ContainerStarted","Data":"6344cc2762d6d91e438957c7a36252609b0764544f08fce6dbcd358d65a48fe5"} Jul 05 19:30:03.482168 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:03.482146 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xnmxp" event={"ID":"5c96e505-de6a-4853-822c-e80de519e002","Type":"ContainerStarted","Data":"8897d37ecfaa968350d5eea450d439f0184d6778ef937586a894ef98ad185753"} Jul 05 19:30:03.483440 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:03.483389 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" event={"ID":"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c","Type":"ContainerStarted","Data":"762468b3eb00b3ff8578b1ae987b7e288f4f107233f0598fe9e58a439d882a0e"} Jul 05 19:30:03.486131 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:03.485625 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-40.ec2.internal" event={"ID":"c667c56910299e2ae0fff31b3d85acc6","Type":"ContainerStarted","Data":"f194830424abc12068299a2c5e23179e77ddd5e169cf14ccf689a9e9acdaac18"} Jul 05 19:30:03.487325 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:03.487301 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal" event={"ID":"1b9cf5d1d2b26f6e5b718a5e66d212ac","Type":"ContainerStarted","Data":"e4c1bb73ebfc54c17db3a674b0172feef17855996651085e75bdd1a0d3ced25d"} Jul 05 19:30:03.515312 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:03.515244 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-40.ec2.internal" podStartSLOduration=3.515224111 podStartE2EDuration="3.515224111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:03.501275861 +0000 UTC m=+4.501293967" watchObservedRunningTime="2026-07-05 19:30:03.515224111 +0000 UTC m=+4.515242219" Jul 05 19:30:04.496671 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:04.496630 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ktlfv" event={"ID":"dc4c3262-1cc8-4aea-a843-26a51eb356f8","Type":"ContainerStarted","Data":"7b510fc394051115a1fed6106729d1f504f1a16f6d953b05d47a48a0cd8e93bd"} Jul 05 19:30:04.501606 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:04.501575 2573 generic.go:356] "Generic (PLEG): container finished" podID="1b9cf5d1d2b26f6e5b718a5e66d212ac" containerID="e4c1bb73ebfc54c17db3a674b0172feef17855996651085e75bdd1a0d3ced25d" exitCode=0 Jul 05 19:30:04.502445 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:04.502409 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal" event={"ID":"1b9cf5d1d2b26f6e5b718a5e66d212ac","Type":"ContainerDied","Data":"e4c1bb73ebfc54c17db3a674b0172feef17855996651085e75bdd1a0d3ced25d"} Jul 05 19:30:04.532672 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:04.532611 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ktlfv" podStartSLOduration=3.47150381 podStartE2EDuration="5.532588782s" totalImagesPullingTime="2.061084972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:04.517226673 +0000 UTC m=+5.517244782" watchObservedRunningTime="2026-07-05 19:30:04.532588782 +0000 UTC m=+5.532606890" Jul 05 19:30:05.044785 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:05.044747 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:05.045107 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:05.045001 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:05.045107 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:05.045084 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret podName:d84a2165-5809-40c0-a5b3-7d3a976291ee nodeName:}" failed. No retries permitted until 2026-07-05 19:30:09.045065912 +0000 UTC m=+10.045084002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret") pod "global-pull-secret-syncer-pcsv4" (UID: "d84a2165-5809-40c0-a5b3-7d3a976291ee") : object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:05.146206 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:05.146170 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs\") pod \"network-metrics-daemon-86bxp\" (UID: \"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3\") " pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:05.146414 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:05.146222 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8b\" (UniqueName: \"kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b\") pod \"network-check-target-qsk8d\" (UID: \"de14a84c-5bb0-4f31-9894-c71098e82aec\") " pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:05.146414 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:05.146367 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 19:30:05.146414 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:05.146388 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 19:30:05.146414 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:05.146401 2573 projected.go:196] Error preparing data for projected volume kube-api-access-h4j8b for pod openshift-network-diagnostics/network-check-target-qsk8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:05.146673 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:05.146462 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b podName:de14a84c-5bb0-4f31-9894-c71098e82aec nodeName:}" failed. No retries permitted until 2026-07-05 19:30:09.146443463 +0000 UTC m=+10.146461550 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4j8b" (UniqueName: "kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b") pod "network-check-target-qsk8d" (UID: "de14a84c-5bb0-4f31-9894-c71098e82aec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:05.146879 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:05.146861 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:05.146964 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:05.146926 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs podName:ed74973b-94ab-43a3-a0d3-b5e89daf5cf3 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:09.146909271 +0000 UTC m=+10.146927360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs") pod "network-metrics-daemon-86bxp" (UID: "ed74973b-94ab-43a3-a0d3-b5e89daf5cf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:05.466272 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:05.466179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:05.466272 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:05.466197 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:05.466272 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:05.466180 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:05.466516 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:05.466490 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:05.469388 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:05.467061 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:05.469388 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:05.467190 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:07.463225 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:07.463188 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:07.463678 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:07.463308 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:07.463678 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:07.463673 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:07.463764 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:07.463743 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:07.463796 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:07.463782 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:07.463851 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:07.463837 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:07.545114 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:07.545079 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ktlfv" Jul 05 19:30:07.545813 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:07.545792 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ktlfv" Jul 05 19:30:08.511213 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:08.511187 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ktlfv" Jul 05 19:30:08.512151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:08.511964 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ktlfv" Jul 05 19:30:09.080609 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:09.080571 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:09.080812 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:09.080718 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:09.080812 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:09.080795 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret podName:d84a2165-5809-40c0-a5b3-7d3a976291ee nodeName:}" failed. No retries permitted until 2026-07-05 19:30:17.080775813 +0000 UTC m=+18.080793905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret") pod "global-pull-secret-syncer-pcsv4" (UID: "d84a2165-5809-40c0-a5b3-7d3a976291ee") : object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:09.181152 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:09.181115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs\") pod \"network-metrics-daemon-86bxp\" (UID: \"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3\") " pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:09.181152 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:09.181156 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8b\" (UniqueName: \"kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b\") pod \"network-check-target-qsk8d\" (UID: \"de14a84c-5bb0-4f31-9894-c71098e82aec\") " pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:09.181387 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:09.181281 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:09.181387 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:09.181365 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs podName:ed74973b-94ab-43a3-a0d3-b5e89daf5cf3 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:17.181340806 +0000 UTC m=+18.181358895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs") pod "network-metrics-daemon-86bxp" (UID: "ed74973b-94ab-43a3-a0d3-b5e89daf5cf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:09.181387 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:09.181285 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 19:30:09.181546 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:09.181402 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 19:30:09.181546 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:09.181416 2573 projected.go:196] Error preparing data for projected volume kube-api-access-h4j8b for pod openshift-network-diagnostics/network-check-target-qsk8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:09.181546 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:09.181467 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b podName:de14a84c-5bb0-4f31-9894-c71098e82aec nodeName:}" failed. No retries permitted until 2026-07-05 19:30:17.181450281 +0000 UTC m=+18.181468368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4j8b" (UniqueName: "kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b") pod "network-check-target-qsk8d" (UID: "de14a84c-5bb0-4f31-9894-c71098e82aec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:09.464088 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:09.463993 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:09.464088 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:09.464000 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:09.465215 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:09.464944 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:09.465215 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:09.465032 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:09.465215 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:09.465098 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:09.465215 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:09.465180 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:11.463183 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:11.463152 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:11.463709 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:11.463266 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:11.463709 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:11.463687 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:11.463827 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:11.463804 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:11.463890 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:11.463881 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:11.463993 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:11.463965 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:13.463939 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:13.463162 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:13.463939 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:13.463177 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:13.463939 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:13.463526 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:13.463939 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:13.463610 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:13.463939 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:13.463175 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:13.463939 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:13.463716 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:13.528625 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:13.528559 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" event={"ID":"1380926c-7f7f-426e-8b55-b3015f6c9ad2","Type":"ContainerStarted","Data":"a546c6369581c095553e97fea7a763447f7107fd8f1364ccd9b6496462f56f58"} Jul 05 19:30:13.530462 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:13.530429 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xnmxp" event={"ID":"5c96e505-de6a-4853-822c-e80de519e002","Type":"ContainerStarted","Data":"473193d87101826abd2f4d718bf7fd97e18ec98bdc2f96c473d03832bb645152"} Jul 05 19:30:13.533498 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:13.533186 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal" event={"ID":"1b9cf5d1d2b26f6e5b718a5e66d212ac","Type":"ContainerStarted","Data":"7b05e866b72e9a238044c8b3ffff4516247e26618c79c99d3126ea59b0e47d61"} Jul 05 19:30:13.535513 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:13.535487 2573 generic.go:356] "Generic (PLEG): container finished" podID="2b2a331e-da1e-4db5-ac53-84d90da88ff3" containerID="72744353f812941dc7df87e54bdcab1e52bfedb97269cbb02712cbbcf06aefc9" exitCode=0 Jul 05 19:30:13.535636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:13.535573 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v428c" event={"ID":"2b2a331e-da1e-4db5-ac53-84d90da88ff3","Type":"ContainerDied","Data":"72744353f812941dc7df87e54bdcab1e52bfedb97269cbb02712cbbcf06aefc9"} Jul 05 19:30:13.538723 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:13.538674 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4snds" event={"ID":"cb2e6493-927f-48b2-8e47-16dc2379911a","Type":"ContainerStarted","Data":"2ae84023a0ab4c3327c573ead86566a8e3275be684d5628d032bfa676d3f2670"} Jul 05 19:30:13.541269 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:13.541241 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" event={"ID":"9536cff9-3d7f-415b-aa0a-8a71113511d8","Type":"ContainerStarted","Data":"66394b4f30d8a6c2e341aa73583488a1b5a2d569c794d3006451f9a974fc9753"} Jul 05 19:30:13.545724 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:13.545668 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xvdjm" podStartSLOduration=4.979731471 podStartE2EDuration="14.545651073s" totalImagesPullingTime="9.565919602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:13.545062722 +0000 UTC m=+14.545080826" watchObservedRunningTime="2026-07-05 19:30:13.545651073 +0000 UTC m=+14.545669181" Jul 05 19:30:13.581340 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:13.581286 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-40.ec2.internal" podStartSLOduration=13.581261543 podStartE2EDuration="13.581261543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:13.560355519 +0000 UTC m=+14.560373626" watchObservedRunningTime="2026-07-05 19:30:13.581261543 +0000 UTC m=+14.581279650" Jul 05 19:30:13.599541 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:13.599437 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xnmxp" podStartSLOduration=5.020183613 podStartE2EDuration="14.599415027s" totalImagesPullingTime="9.579231414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:13.581756942 +0000 UTC m=+14.581775047" watchObservedRunningTime="2026-07-05 19:30:13.599415027 +0000 UTC m=+14.599433135" Jul 05 19:30:13.622111 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:13.622027 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4snds" podStartSLOduration=5.054139457 podStartE2EDuration="14.622006085s" totalImagesPullingTime="9.567866628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:13.599819659 +0000 UTC m=+14.599837764" watchObservedRunningTime="2026-07-05 19:30:13.622006085 +0000 UTC m=+14.622024193" Jul 05 19:30:14.544904 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:14.544839 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jx5q9" event={"ID":"a9d9c89b-7f73-4ca0-a000-1d01ab3ec580","Type":"ContainerStarted","Data":"1f6b7666551c0df2dc5f173552777caee662374741a57410d6af17dcacf6e0f0"} Jul 05 19:30:14.560382 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:14.560324 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jx5q9" podStartSLOduration=5.509906491 podStartE2EDuration="15.560303565s" totalImagesPullingTime="10.050397074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:14.559883442 +0000 UTC m=+15.559901548" watchObservedRunningTime="2026-07-05 19:30:14.560303565 +0000 UTC m=+15.560321672" Jul 05 19:30:15.462965 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:15.462922 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:15.463174 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:15.462996 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:15.463174 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:15.463010 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:15.463174 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:15.463140 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:15.463291 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:15.463220 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:15.463334 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:15.463314 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:17.152422 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:17.152243 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:17.152422 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:17.152369 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:17.153179 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:17.152454 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret podName:d84a2165-5809-40c0-a5b3-7d3a976291ee nodeName:}" failed. No retries permitted until 2026-07-05 19:30:33.152434613 +0000 UTC m=+34.152452715 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret") pod "global-pull-secret-syncer-pcsv4" (UID: "d84a2165-5809-40c0-a5b3-7d3a976291ee") : object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:17.253268 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:17.253225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8b\" (UniqueName: \"kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b\") pod \"network-check-target-qsk8d\" (UID: \"de14a84c-5bb0-4f31-9894-c71098e82aec\") " pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:17.253431 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:17.253330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs\") pod \"network-metrics-daemon-86bxp\" (UID: \"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3\") " pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:17.253431 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:17.253411 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 19:30:17.253519 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:17.253435 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:17.253519 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:17.253442 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 19:30:17.253519 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:17.253456 2573 projected.go:196] Error preparing data for projected volume kube-api-access-h4j8b for pod openshift-network-diagnostics/network-check-target-qsk8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:17.253519 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:17.253501 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs podName:ed74973b-94ab-43a3-a0d3-b5e89daf5cf3 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:33.253480806 +0000 UTC m=+34.253498896 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs") pod "network-metrics-daemon-86bxp" (UID: "ed74973b-94ab-43a3-a0d3-b5e89daf5cf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:17.253519 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:17.253521 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b podName:de14a84c-5bb0-4f31-9894-c71098e82aec nodeName:}" failed. No retries permitted until 2026-07-05 19:30:33.253511697 +0000 UTC m=+34.253529785 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4j8b" (UniqueName: "kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b") pod "network-check-target-qsk8d" (UID: "de14a84c-5bb0-4f31-9894-c71098e82aec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:17.462787 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:17.462706 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:17.462787 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:17.462725 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:17.462993 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:17.462706 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:17.462993 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:17.462819 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:17.462993 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:17.462903 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:17.463187 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:17.463025 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:19.464613 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:19.464569 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:19.465152 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:19.464690 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:19.465152 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:19.464686 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:19.465152 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:19.464781 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:19.465152 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:19.464899 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:19.465152 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:19.464985 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:21.463692 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:21.463660 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:21.464157 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:21.463660 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:21.464157 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:21.463796 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:21.464157 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:21.463807 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:21.464157 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:21.463888 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:21.464157 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:21.463979 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:21.605452 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:21.605114 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 05 19:30:22.480288 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:22.479835 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-05T19:30:21.605257651Z","UUID":"1c6a1ef0-2beb-4645-b649-7d63a836fd66","Handler":null,"Name":"","Endpoint":""} Jul 05 19:30:22.481487 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:22.481453 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 05 19:30:22.481605 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:22.481497 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 05 19:30:22.564194 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:22.564157 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" event={"ID":"9536cff9-3d7f-415b-aa0a-8a71113511d8","Type":"ContainerStarted","Data":"2cb6b93d52c13c39f5107910c606caa3fea6f30dd61721ae0b4432fe7850dec9"} Jul 05 19:30:22.565471 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:22.565440 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l7gg4" event={"ID":"cfaa8adb-2377-43ab-8481-f3d30e66a994","Type":"ContainerStarted","Data":"738fdd72bc5486a9deb1c94423704754e813a8ea2698795794ed16ec4d319d7b"} Jul 05 19:30:22.568198 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:22.568178 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" event={"ID":"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c","Type":"ContainerStarted","Data":"9d2c26f240bd5772d050c734fde9f5ae55a45998aca833ebb2172b8ef2917f37"} Jul 05 19:30:22.568276 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:22.568240 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" event={"ID":"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c","Type":"ContainerStarted","Data":"4345c832d5bab280a0dc8ace098457b921dd2966df050fd6d59aca39d75345d5"} Jul 05 19:30:22.568276 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:22.568273 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" event={"ID":"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c","Type":"ContainerStarted","Data":"689a897d5fa16b6227162f072b4136fa1d9f1c50b205889c0eff616be84609b0"} Jul 05 19:30:22.568346 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:22.568300 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" event={"ID":"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c","Type":"ContainerStarted","Data":"c68f73bea85dd9a076f3948df079e8a0d64436b203526ea4e199f2b7d8e8586e"} Jul 05 19:30:22.568346 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:22.568327 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" event={"ID":"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c","Type":"ContainerStarted","Data":"22a69f69a73da317a63080ea422abc93944ea27098eed2431def87490e4d301a"} Jul 05 19:30:22.568400 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:22.568355 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" event={"ID":"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c","Type":"ContainerStarted","Data":"60923bd4ed3f4e443d7c351f2649444a186bdaadf416021cd4dfb9ac93c67c26"} Jul 05 19:30:22.582138 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:22.582098 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l7gg4" podStartSLOduration=4.945690193 podStartE2EDuration="23.58208486s" totalImagesPullingTime="18.636394667s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:22.581907941 +0000 UTC m=+23.581926047" watchObservedRunningTime="2026-07-05 19:30:22.58208486 +0000 UTC m=+23.582102966" Jul 05 19:30:23.463279 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:23.463204 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:23.463279 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:23.463240 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:23.463513 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:23.463205 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:23.463513 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:23.463301 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:23.463513 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:23.463393 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:23.463513 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:23.463484 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:23.572670 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:23.572635 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" event={"ID":"9536cff9-3d7f-415b-aa0a-8a71113511d8","Type":"ContainerStarted","Data":"b4a59823a996b3f9fd94d57b7feaf5f123d6dc7a38ae2e60bb5293039bac882d"} Jul 05 19:30:23.590715 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:23.590669 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-smlfw" podStartSLOduration=4.714920468 podStartE2EDuration="24.590654628s" totalImagesPullingTime="19.87573416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:29:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:23.590555869 +0000 UTC m=+24.590573977" watchObservedRunningTime="2026-07-05 19:30:23.590654628 +0000 UTC m=+24.590672730" Jul 05 19:30:24.579382 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:24.579339 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" event={"ID":"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c","Type":"ContainerStarted","Data":"087e8f667aca58acafd4a3b63970a312fc041ac661234fe7587b6766ac383edd"} Jul 05 19:30:25.463060 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:25.463007 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:25.463256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:25.463007 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:25.463256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:25.463025 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:25.463256 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:25.463199 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:25.463478 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:25.463268 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:25.463478 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:25.463152 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:27.463459 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:27.463273 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:27.464160 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:27.463273 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:27.464160 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:27.463559 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:27.464160 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:27.463312 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:27.464160 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:27.463613 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:27.464160 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:27.463740 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:27.590397 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:27.590354 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" event={"ID":"46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c","Type":"ContainerStarted","Data":"be416fe7d199129962d0dc8f56283435940bcc79c141be0a349d0ced64186704"} Jul 05 19:30:27.590700 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:27.590666 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:27.592571 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:27.592548 2573 generic.go:356] "Generic (PLEG): container finished" podID="2b2a331e-da1e-4db5-ac53-84d90da88ff3" containerID="84efc7fd694257a79319d83bb336a6b3bb75da391b709a7ab06963b5ae662ead" exitCode=0 Jul 05 19:30:27.592675 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:27.592590 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v428c" event={"ID":"2b2a331e-da1e-4db5-ac53-84d90da88ff3","Type":"ContainerDied","Data":"84efc7fd694257a79319d83bb336a6b3bb75da391b709a7ab06963b5ae662ead"} Jul 05 19:30:27.606998 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:27.606978 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:27.621905 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:27.621854 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" podStartSLOduration=9.92550027 podStartE2EDuration="28.621836587s" totalImagesPullingTime="18.696336317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:27.620876337 +0000 UTC m=+28.620894438" watchObservedRunningTime="2026-07-05 19:30:27.621836587 +0000 UTC m=+28.621854694" Jul 05 19:30:28.543948 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:28.543907 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-86bxp"] Jul 05 19:30:28.544472 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:28.544098 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:28.544472 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:28.544223 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:28.547338 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:28.547316 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pcsv4"] Jul 05 19:30:28.547473 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:28.547445 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:28.547574 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:28.547554 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:28.548106 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:28.548084 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qsk8d"] Jul 05 19:30:28.548205 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:28.548168 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:28.548259 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:28.548231 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:28.595560 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:28.595411 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:28.595560 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:28.595503 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:28.613650 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:28.613628 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:30:29.599118 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:29.599078 2573 generic.go:356] "Generic (PLEG): container finished" podID="2b2a331e-da1e-4db5-ac53-84d90da88ff3" containerID="55a1fb039bdbbd3c09511e774fb4859e8034ac7076929f684be9c2fd596442be" exitCode=0 Jul 05 19:30:29.599492 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:29.599164 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v428c" event={"ID":"2b2a331e-da1e-4db5-ac53-84d90da88ff3","Type":"ContainerDied","Data":"55a1fb039bdbbd3c09511e774fb4859e8034ac7076929f684be9c2fd596442be"} Jul 05 19:30:30.463185 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:30.463153 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:30.463309 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:30.463190 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:30.463309 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:30.463190 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:30.463309 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:30.463279 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:30.463471 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:30.463349 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:30.463471 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:30.463406 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:31.605938 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:31.605760 2573 generic.go:356] "Generic (PLEG): container finished" podID="2b2a331e-da1e-4db5-ac53-84d90da88ff3" containerID="0ef48c20c4a7adbeb03a0aefecb8926d6b1bcbbfad64a1df33b838fb166a8a96" exitCode=0 Jul 05 19:30:31.606432 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:31.605844 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v428c" event={"ID":"2b2a331e-da1e-4db5-ac53-84d90da88ff3","Type":"ContainerDied","Data":"0ef48c20c4a7adbeb03a0aefecb8926d6b1bcbbfad64a1df33b838fb166a8a96"} Jul 05 19:30:32.463206 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:32.463179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:32.463392 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:32.463179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:32.463392 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:32.463293 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:32.463392 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:32.463180 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:32.463392 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:32.463360 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:32.463586 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:32.463494 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:33.170663 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:33.170622 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:33.171304 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:33.170789 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:33.171304 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:33.170863 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret podName:d84a2165-5809-40c0-a5b3-7d3a976291ee nodeName:}" failed. No retries permitted until 2026-07-05 19:31:05.170844228 +0000 UTC m=+66.170862329 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret") pod "global-pull-secret-syncer-pcsv4" (UID: "d84a2165-5809-40c0-a5b3-7d3a976291ee") : object "kube-system"/"original-pull-secret" not registered Jul 05 19:30:33.271224 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:33.271189 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs\") pod \"network-metrics-daemon-86bxp\" (UID: \"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3\") " pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:33.271224 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:33.271226 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8b\" (UniqueName: \"kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b\") pod \"network-check-target-qsk8d\" (UID: \"de14a84c-5bb0-4f31-9894-c71098e82aec\") " pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:33.271432 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:33.271332 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:33.271432 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:33.271341 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 19:30:33.271432 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:33.271357 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 19:30:33.271432 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:33.271366 2573 projected.go:196] Error preparing data for projected volume kube-api-access-h4j8b for pod openshift-network-diagnostics/network-check-target-qsk8d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:33.271432 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:33.271410 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs podName:ed74973b-94ab-43a3-a0d3-b5e89daf5cf3 nodeName:}" failed. No retries permitted until 2026-07-05 19:31:05.271391966 +0000 UTC m=+66.271410070 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs") pod "network-metrics-daemon-86bxp" (UID: "ed74973b-94ab-43a3-a0d3-b5e89daf5cf3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 19:30:33.271432 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:33.271426 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b podName:de14a84c-5bb0-4f31-9894-c71098e82aec nodeName:}" failed. No retries permitted until 2026-07-05 19:31:05.271419543 +0000 UTC m=+66.271437626 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4j8b" (UniqueName: "kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b") pod "network-check-target-qsk8d" (UID: "de14a84c-5bb0-4f31-9894-c71098e82aec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 19:30:34.463329 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:34.463292 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:34.463772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:34.463355 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:34.463772 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:34.463459 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:34.463772 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:34.463477 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qsk8d" podUID="de14a84c-5bb0-4f31-9894-c71098e82aec" Jul 05 19:30:34.463772 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:34.463546 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-86bxp" podUID="ed74973b-94ab-43a3-a0d3-b5e89daf5cf3" Jul 05 19:30:34.463772 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:34.463616 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcsv4" podUID="d84a2165-5809-40c0-a5b3-7d3a976291ee" Jul 05 19:30:35.345149 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.345115 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-40.ec2.internal" event="NodeReady" Jul 05 19:30:35.345329 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.345275 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 05 19:30:35.379420 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.379393 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-wz8gp"] Jul 05 19:30:35.383279 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.383251 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf"] Jul 05 19:30:35.383568 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.383543 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.386846 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.386825 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7575585c8f-8gvjn"] Jul 05 19:30:35.387736 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.386962 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:30:35.387736 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.387511 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 05 19:30:35.387736 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.387628 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zpnrg\"" Jul 05 19:30:35.387914 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.387744 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 05 19:30:35.387914 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.387884 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 05 19:30:35.388205 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.388190 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 05 19:30:35.390677 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.390060 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-9d66c8c9c-mg7q2"] Jul 05 19:30:35.390677 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.390152 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.390677 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.390159 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 05 19:30:35.390677 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.390200 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-trq5g\"" Jul 05 19:30:35.390677 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.390468 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 05 19:30:35.390677 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.390673 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 05 19:30:35.392483 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.392465 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 05 19:30:35.393329 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.393297 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-9rf6c"] Jul 05 19:30:35.393510 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.393493 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:35.395444 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.395419 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 05 19:30:35.395540 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.395459 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k29fx\"" Jul 05 19:30:35.395540 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.395466 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 05 19:30:35.395540 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.395485 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 05 19:30:35.397208 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.397184 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-qb5w5"] Jul 05 19:30:35.400687 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.400497 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-9rf6c" Jul 05 19:30:35.404727 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.402717 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 05 19:30:35.406520 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.406500 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 05 19:30:35.406733 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.406717 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 05 19:30:35.406928 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.406913 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 05 19:30:35.407169 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.407152 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 05 19:30:35.407875 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.407857 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 05 19:30:35.409706 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.408431 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cghrd\"" Jul 05 19:30:35.409808 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.409792 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 05 19:30:35.409870 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.408872 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 05 19:30:35.409923 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.409163 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 05 19:30:35.412122 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.412069 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j"] Jul 05 19:30:35.412287 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.412243 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:35.415871 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.415836 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-bwfrs\"" Jul 05 19:30:35.415967 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.415880 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 05 19:30:35.415967 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.415903 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 05 19:30:35.417028 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.416983 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 05 19:30:35.417277 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.417252 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 05 19:30:35.417513 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.417492 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz"] Jul 05 19:30:35.417816 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.417799 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" Jul 05 19:30:35.418304 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.418282 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-z4w4r\"" Jul 05 19:30:35.419112 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.419096 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 05 19:30:35.421630 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.421605 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 05 19:30:35.422466 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.422429 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-krqrz\"" Jul 05 19:30:35.422719 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.422701 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 05 19:30:35.423771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.423748 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 05 19:30:35.425102 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.425084 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp"] Jul 05 19:30:35.426127 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.426099 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" Jul 05 19:30:35.427230 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.427210 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 05 19:30:35.428749 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.428731 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 05 19:30:35.428908 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.428889 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 05 19:30:35.429209 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.429173 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5rdjr\"" Jul 05 19:30:35.429209 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.429203 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 05 19:30:35.429379 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.429356 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 05 19:30:35.429815 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.429792 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-wz8gp"] Jul 05 19:30:35.429905 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.429835 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-9rf6c"] Jul 05 19:30:35.429905 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.429860 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-hx424"] Jul 05 19:30:35.430273 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.430252 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" Jul 05 19:30:35.432867 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.432688 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-g4tmc\"" Jul 05 19:30:35.432867 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.432702 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 05 19:30:35.432867 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.432718 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 05 19:30:35.433073 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.432993 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 05 19:30:35.433131 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.433120 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 05 19:30:35.433385 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.433369 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf"] Jul 05 19:30:35.433500 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.433484 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-9d66c8c9c-mg7q2"] Jul 05 19:30:35.433646 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.433512 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j"] Jul 05 19:30:35.433646 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.433535 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7575585c8f-8gvjn"] Jul 05 19:30:35.433646 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.433548 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-qb5w5"] Jul 05 19:30:35.433646 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.433569 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6vhql"] Jul 05 19:30:35.433646 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.433487 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hx424" Jul 05 19:30:35.436330 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.436309 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 05 19:30:35.436429 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.436381 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-lt62k\"" Jul 05 19:30:35.436541 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.436521 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 05 19:30:35.436779 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.436762 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4vprg"] Jul 05 19:30:35.436964 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.436945 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:35.440758 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.440737 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 05 19:30:35.440862 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.440742 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 05 19:30:35.441164 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.441147 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nkl9h\"" Jul 05 19:30:35.441268 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.441251 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp"] Jul 05 19:30:35.441330 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.441286 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz"] Jul 05 19:30:35.441330 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.441319 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-hx424"] Jul 05 19:30:35.441431 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.441336 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4vprg"] Jul 05 19:30:35.441431 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.441352 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6vhql"] Jul 05 19:30:35.441528 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.441482 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4vprg" Jul 05 19:30:35.444107 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.444064 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 05 19:30:35.444217 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.444202 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8zj4k\"" Jul 05 19:30:35.444273 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.444228 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 05 19:30:35.445667 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.445650 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 05 19:30:35.487177 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-default-certificate\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:35.487636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487179 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:35.487636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487207 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:35.487636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487236 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc38af11-062b-458d-8f23-4d34f9fd1fdd-tmp\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.487636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487261 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlfcd\" (UniqueName: \"kubernetes.io/projected/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-kube-api-access-wlfcd\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:35.487636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487292 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/90a56a02-5f5d-48bd-9285-c4db913af929-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-xcnwf\" (UID: \"90a56a02-5f5d-48bd-9285-c4db913af929\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:30:35.487636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487315 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cab1cf3-0844-475a-9a85-45b7621f7be0-ca-trust-extracted\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.487636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487337 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppgqz\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-kube-api-access-ppgqz\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.487636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487417 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cab1cf3-0844-475a-9a85-45b7621f7be0-trusted-ca\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.487636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487453 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6d4t\" (UniqueName: \"kubernetes.io/projected/bc38af11-062b-458d-8f23-4d34f9fd1fdd-kube-api-access-t6d4t\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.487636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487478 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxrx\" (UniqueName: \"kubernetes.io/projected/619c0eb1-3880-4758-b423-9acf269979c0-kube-api-access-dpxrx\") pod \"volume-data-source-validator-5d456fdbcd-9rf6c\" (UID: \"619c0eb1-3880-4758-b423-9acf269979c0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-9rf6c" Jul 05 19:30:35.487636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487515 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-certificates\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.487636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487539 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cab1cf3-0844-475a-9a85-45b7621f7be0-installation-pull-secrets\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.487636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487564 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-stats-auth\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:35.487636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487617 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc38af11-062b-458d-8f23-4d34f9fd1fdd-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.487636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487641 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc38af11-062b-458d-8f23-4d34f9fd1fdd-serving-cert\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.488254 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487665 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffpqj\" (UniqueName: \"kubernetes.io/projected/90a56a02-5f5d-48bd-9285-c4db913af929-kube-api-access-ffpqj\") pod \"cluster-monitoring-operator-64786d9cbb-xcnwf\" (UID: \"90a56a02-5f5d-48bd-9285-c4db913af929\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:30:35.488254 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487690 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cab1cf3-0844-475a-9a85-45b7621f7be0-image-registry-private-configuration\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.488254 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487840 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bc38af11-062b-458d-8f23-4d34f9fd1fdd-snapshots\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.488254 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487899 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xcnwf\" (UID: \"90a56a02-5f5d-48bd-9285-c4db913af929\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:30:35.488254 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487936 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.488254 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.487971 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-bound-sa-token\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.488254 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.488007 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc38af11-062b-458d-8f23-4d34f9fd1fdd-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.588999 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.588956 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85f8e787-c095-491d-91e6-2840a6364447-cert\") pod \"ingress-canary-4vprg\" (UID: \"85f8e787-c095-491d-91e6-2840a6364447\") " pod="openshift-ingress-canary/ingress-canary-4vprg" Jul 05 19:30:35.589201 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589009 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnq8h\" (UniqueName: \"kubernetes.io/projected/9d64b827-c0f3-4a50-8e75-6284cffcf625-kube-api-access-bnq8h\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:35.589201 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589035 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9qg7\" (UniqueName: \"kubernetes.io/projected/39d68362-0254-4bd3-925d-cd2c5767a5d8-kube-api-access-z9qg7\") pod \"cluster-samples-operator-56fcf86bf-cj69j\" (UID: \"39d68362-0254-4bd3-925d-cd2c5767a5d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" Jul 05 19:30:35.589201 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-certificates\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.589201 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-stats-auth\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:35.589201 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589152 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81eba20f-cd1e-41cb-a868-ee8f133a20ef-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-k8pzp\" (UID: \"81eba20f-cd1e-41cb-a868-ee8f133a20ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" Jul 05 19:30:35.589201 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589179 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bc38af11-062b-458d-8f23-4d34f9fd1fdd-snapshots\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.589438 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589205 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xcnwf\" (UID: \"90a56a02-5f5d-48bd-9285-c4db913af929\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:30:35.589438 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589273 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cab1cf3-0844-475a-9a85-45b7621f7be0-image-registry-private-configuration\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.589438 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589314 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.589438 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/81eba20f-cd1e-41cb-a868-ee8f133a20ef-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-k8pzp\" (UID: \"81eba20f-cd1e-41cb-a868-ee8f133a20ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" Jul 05 19:30:35.589438 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589380 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlfcd\" (UniqueName: \"kubernetes.io/projected/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-kube-api-access-wlfcd\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:35.589438 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlr5d\" (UniqueName: \"kubernetes.io/projected/f667ec66-87f7-4789-9288-875d41952cc4-kube-api-access-jlr5d\") pod \"network-check-source-989f8b8dc-hx424\" (UID: \"f667ec66-87f7-4789-9288-875d41952cc4\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hx424" Jul 05 19:30:35.589627 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589439 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cab1cf3-0844-475a-9a85-45b7621f7be0-ca-trust-extracted\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.589627 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589464 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppgqz\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-kube-api-access-ppgqz\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.589627 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589492 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:35.589627 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589519 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-cj69j\" (UID: \"39d68362-0254-4bd3-925d-cd2c5767a5d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" Jul 05 19:30:35.589627 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589553 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntj5r\" (UniqueName: \"kubernetes.io/projected/81eba20f-cd1e-41cb-a868-ee8f133a20ef-kube-api-access-ntj5r\") pod \"kube-storage-version-migrator-operator-846bf668f7-k8pzp\" (UID: \"81eba20f-cd1e-41cb-a868-ee8f133a20ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" Jul 05 19:30:35.589627 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589594 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6d4t\" (UniqueName: \"kubernetes.io/projected/bc38af11-062b-458d-8f23-4d34f9fd1fdd-kube-api-access-t6d4t\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.589627 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589617 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxrx\" (UniqueName: \"kubernetes.io/projected/619c0eb1-3880-4758-b423-9acf269979c0-kube-api-access-dpxrx\") pod \"volume-data-source-validator-5d456fdbcd-9rf6c\" (UID: \"619c0eb1-3880-4758-b423-9acf269979c0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-9rf6c" Jul 05 19:30:35.589927 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589646 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xjz8\" (UniqueName: \"kubernetes.io/projected/1071794b-4f80-4a07-a571-e790c329a568-kube-api-access-7xjz8\") pod \"service-ca-operator-5d46d9ffd4-tn8sz\" (UID: \"1071794b-4f80-4a07-a571-e790c329a568\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" Jul 05 19:30:35.589927 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589671 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr9xg\" (UniqueName: \"kubernetes.io/projected/85f8e787-c095-491d-91e6-2840a6364447-kube-api-access-hr9xg\") pod \"ingress-canary-4vprg\" (UID: \"85f8e787-c095-491d-91e6-2840a6364447\") " pod="openshift-ingress-canary/ingress-canary-4vprg" Jul 05 19:30:35.589927 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589710 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d50d6beb-5c77-4e3e-a170-a9220239ce86-config\") pod \"console-operator-68b757fb67-qb5w5\" (UID: \"d50d6beb-5c77-4e3e-a170-a9220239ce86\") " pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:35.589927 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589740 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-certificates\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.589927 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589826 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bc38af11-062b-458d-8f23-4d34f9fd1fdd-snapshots\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.589927 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589909 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cab1cf3-0844-475a-9a85-45b7621f7be0-installation-pull-secrets\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.590228 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.589999 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cab1cf3-0844-475a-9a85-45b7621f7be0-ca-trust-extracted\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.590228 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.590073 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:35.590228 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.590100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d64b827-c0f3-4a50-8e75-6284cffcf625-tmp-dir\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:35.590228 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:35.590172 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 05 19:30:35.590427 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:35.590248 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 05 19:30:35.590427 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:35.590259 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs podName:9d9bd347-ceb3-4b36-8654-7f2ddbecf9af nodeName:}" failed. No retries permitted until 2026-07-05 19:30:36.090237992 +0000 UTC m=+37.090256077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs") pod "router-default-9d66c8c9c-mg7q2" (UID: "9d9bd347-ceb3-4b36-8654-7f2ddbecf9af") : secret "router-metrics-certs-default" not found Jul 05 19:30:35.590427 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:35.590312 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 19:30:35.590427 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.590314 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1071794b-4f80-4a07-a571-e790c329a568-config\") pod \"service-ca-operator-5d46d9ffd4-tn8sz\" (UID: \"1071794b-4f80-4a07-a571-e790c329a568\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" Jul 05 19:30:35.590427 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:35.590323 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7575585c8f-8gvjn: secret "image-registry-tls" not found Jul 05 19:30:35.590427 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.590367 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc38af11-062b-458d-8f23-4d34f9fd1fdd-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.590427 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:35.590385 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls podName:90a56a02-5f5d-48bd-9285-c4db913af929 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:36.090370377 +0000 UTC m=+37.090388464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-xcnwf" (UID: "90a56a02-5f5d-48bd-9285-c4db913af929") : secret "cluster-monitoring-operator-tls" not found Jul 05 19:30:35.590427 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.590411 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc38af11-062b-458d-8f23-4d34f9fd1fdd-serving-cert\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.590836 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.590438 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffpqj\" (UniqueName: \"kubernetes.io/projected/90a56a02-5f5d-48bd-9285-c4db913af929-kube-api-access-ffpqj\") pod \"cluster-monitoring-operator-64786d9cbb-xcnwf\" (UID: \"90a56a02-5f5d-48bd-9285-c4db913af929\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:30:35.590836 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:35.590492 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls podName:8cab1cf3-0844-475a-9a85-45b7621f7be0 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:36.090480329 +0000 UTC m=+37.090498418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls") pod "image-registry-7575585c8f-8gvjn" (UID: "8cab1cf3-0844-475a-9a85-45b7621f7be0") : secret "image-registry-tls" not found Jul 05 19:30:35.590836 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.590517 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:35.590836 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.590541 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-bound-sa-token\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.590836 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.590567 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1071794b-4f80-4a07-a571-e790c329a568-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-tn8sz\" (UID: \"1071794b-4f80-4a07-a571-e790c329a568\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" Jul 05 19:30:35.590836 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.590591 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d50d6beb-5c77-4e3e-a170-a9220239ce86-serving-cert\") pod \"console-operator-68b757fb67-qb5w5\" (UID: \"d50d6beb-5c77-4e3e-a170-a9220239ce86\") " pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:35.590836 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.590643 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbqhp\" (UniqueName: \"kubernetes.io/projected/d50d6beb-5c77-4e3e-a170-a9220239ce86-kube-api-access-jbqhp\") pod \"console-operator-68b757fb67-qb5w5\" (UID: \"d50d6beb-5c77-4e3e-a170-a9220239ce86\") " pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:35.590836 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.590670 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc38af11-062b-458d-8f23-4d34f9fd1fdd-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.590836 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.590695 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-default-certificate\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:35.591303 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:35.591099 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle podName:9d9bd347-ceb3-4b36-8654-7f2ddbecf9af nodeName:}" failed. No retries permitted until 2026-07-05 19:30:36.091016854 +0000 UTC m=+37.091034939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle") pod "router-default-9d66c8c9c-mg7q2" (UID: "9d9bd347-ceb3-4b36-8654-7f2ddbecf9af") : configmap references non-existent config key: service-ca.crt Jul 05 19:30:35.591303 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.591274 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc38af11-062b-458d-8f23-4d34f9fd1fdd-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.591796 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.591660 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc38af11-062b-458d-8f23-4d34f9fd1fdd-tmp\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.591796 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.591744 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc38af11-062b-458d-8f23-4d34f9fd1fdd-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.591796 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.591745 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/90a56a02-5f5d-48bd-9285-c4db913af929-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-xcnwf\" (UID: \"90a56a02-5f5d-48bd-9285-c4db913af929\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:30:35.593353 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.592084 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d64b827-c0f3-4a50-8e75-6284cffcf625-config-volume\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:35.593353 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.592118 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d50d6beb-5c77-4e3e-a170-a9220239ce86-trusted-ca\") pod \"console-operator-68b757fb67-qb5w5\" (UID: \"d50d6beb-5c77-4e3e-a170-a9220239ce86\") " pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:35.593353 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.592215 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cab1cf3-0844-475a-9a85-45b7621f7be0-trusted-ca\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.593353 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.592485 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc38af11-062b-458d-8f23-4d34f9fd1fdd-tmp\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.593353 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.593158 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cab1cf3-0844-475a-9a85-45b7621f7be0-trusted-ca\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.593353 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.593296 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/90a56a02-5f5d-48bd-9285-c4db913af929-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-xcnwf\" (UID: \"90a56a02-5f5d-48bd-9285-c4db913af929\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:30:35.595062 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.595015 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc38af11-062b-458d-8f23-4d34f9fd1fdd-serving-cert\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.595235 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.595132 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-stats-auth\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:35.595412 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.595382 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cab1cf3-0844-475a-9a85-45b7621f7be0-installation-pull-secrets\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.595612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.595595 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-default-certificate\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:35.595669 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.595634 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cab1cf3-0844-475a-9a85-45b7621f7be0-image-registry-private-configuration\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.599614 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.599542 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlfcd\" (UniqueName: \"kubernetes.io/projected/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-kube-api-access-wlfcd\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:35.599835 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.599792 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffpqj\" (UniqueName: \"kubernetes.io/projected/90a56a02-5f5d-48bd-9285-c4db913af929-kube-api-access-ffpqj\") pod \"cluster-monitoring-operator-64786d9cbb-xcnwf\" (UID: \"90a56a02-5f5d-48bd-9285-c4db913af929\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:30:35.599943 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.599904 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-bound-sa-token\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.600156 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.600137 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxrx\" (UniqueName: \"kubernetes.io/projected/619c0eb1-3880-4758-b423-9acf269979c0-kube-api-access-dpxrx\") pod \"volume-data-source-validator-5d456fdbcd-9rf6c\" (UID: \"619c0eb1-3880-4758-b423-9acf269979c0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-9rf6c" Jul 05 19:30:35.600234 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.600158 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6d4t\" (UniqueName: \"kubernetes.io/projected/bc38af11-062b-458d-8f23-4d34f9fd1fdd-kube-api-access-t6d4t\") pod \"insights-operator-5c847dcfc4-wz8gp\" (UID: \"bc38af11-062b-458d-8f23-4d34f9fd1fdd\") " pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.600778 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.600756 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppgqz\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-kube-api-access-ppgqz\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:35.693621 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.693567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1071794b-4f80-4a07-a571-e790c329a568-config\") pod \"service-ca-operator-5d46d9ffd4-tn8sz\" (UID: \"1071794b-4f80-4a07-a571-e790c329a568\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" Jul 05 19:30:35.693621 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.693631 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1071794b-4f80-4a07-a571-e790c329a568-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-tn8sz\" (UID: \"1071794b-4f80-4a07-a571-e790c329a568\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" Jul 05 19:30:35.693876 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.693648 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d50d6beb-5c77-4e3e-a170-a9220239ce86-serving-cert\") pod \"console-operator-68b757fb67-qb5w5\" (UID: \"d50d6beb-5c77-4e3e-a170-a9220239ce86\") " pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:35.693876 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.693668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbqhp\" (UniqueName: \"kubernetes.io/projected/d50d6beb-5c77-4e3e-a170-a9220239ce86-kube-api-access-jbqhp\") pod \"console-operator-68b757fb67-qb5w5\" (UID: \"d50d6beb-5c77-4e3e-a170-a9220239ce86\") " pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:35.693876 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.693691 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d64b827-c0f3-4a50-8e75-6284cffcf625-config-volume\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:35.693876 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.693708 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d50d6beb-5c77-4e3e-a170-a9220239ce86-trusted-ca\") pod \"console-operator-68b757fb67-qb5w5\" (UID: \"d50d6beb-5c77-4e3e-a170-a9220239ce86\") " pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:35.693876 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.693804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85f8e787-c095-491d-91e6-2840a6364447-cert\") pod \"ingress-canary-4vprg\" (UID: \"85f8e787-c095-491d-91e6-2840a6364447\") " pod="openshift-ingress-canary/ingress-canary-4vprg" Jul 05 19:30:35.693876 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.693848 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnq8h\" (UniqueName: \"kubernetes.io/projected/9d64b827-c0f3-4a50-8e75-6284cffcf625-kube-api-access-bnq8h\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:35.693876 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.693875 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9qg7\" (UniqueName: \"kubernetes.io/projected/39d68362-0254-4bd3-925d-cd2c5767a5d8-kube-api-access-z9qg7\") pod \"cluster-samples-operator-56fcf86bf-cj69j\" (UID: \"39d68362-0254-4bd3-925d-cd2c5767a5d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" Jul 05 19:30:35.694239 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.693906 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81eba20f-cd1e-41cb-a868-ee8f133a20ef-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-k8pzp\" (UID: \"81eba20f-cd1e-41cb-a868-ee8f133a20ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" Jul 05 19:30:35.694239 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.693978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/81eba20f-cd1e-41cb-a868-ee8f133a20ef-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-k8pzp\" (UID: \"81eba20f-cd1e-41cb-a868-ee8f133a20ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" Jul 05 19:30:35.694239 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.694017 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlr5d\" (UniqueName: \"kubernetes.io/projected/f667ec66-87f7-4789-9288-875d41952cc4-kube-api-access-jlr5d\") pod \"network-check-source-989f8b8dc-hx424\" (UID: \"f667ec66-87f7-4789-9288-875d41952cc4\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hx424" Jul 05 19:30:35.694239 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.694076 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:35.694239 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.694104 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-cj69j\" (UID: \"39d68362-0254-4bd3-925d-cd2c5767a5d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" Jul 05 19:30:35.694239 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.694137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntj5r\" (UniqueName: \"kubernetes.io/projected/81eba20f-cd1e-41cb-a868-ee8f133a20ef-kube-api-access-ntj5r\") pod \"kube-storage-version-migrator-operator-846bf668f7-k8pzp\" (UID: \"81eba20f-cd1e-41cb-a868-ee8f133a20ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" Jul 05 19:30:35.694239 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.694181 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xjz8\" (UniqueName: \"kubernetes.io/projected/1071794b-4f80-4a07-a571-e790c329a568-kube-api-access-7xjz8\") pod \"service-ca-operator-5d46d9ffd4-tn8sz\" (UID: \"1071794b-4f80-4a07-a571-e790c329a568\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" Jul 05 19:30:35.694239 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.694206 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr9xg\" (UniqueName: \"kubernetes.io/projected/85f8e787-c095-491d-91e6-2840a6364447-kube-api-access-hr9xg\") pod \"ingress-canary-4vprg\" (UID: \"85f8e787-c095-491d-91e6-2840a6364447\") " pod="openshift-ingress-canary/ingress-canary-4vprg" Jul 05 19:30:35.694239 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.694233 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d50d6beb-5c77-4e3e-a170-a9220239ce86-config\") pod \"console-operator-68b757fb67-qb5w5\" (UID: \"d50d6beb-5c77-4e3e-a170-a9220239ce86\") " pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:35.694648 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.694274 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d64b827-c0f3-4a50-8e75-6284cffcf625-tmp-dir\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:35.694648 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.694348 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1071794b-4f80-4a07-a571-e790c329a568-config\") pod \"service-ca-operator-5d46d9ffd4-tn8sz\" (UID: \"1071794b-4f80-4a07-a571-e790c329a568\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" Jul 05 19:30:35.694648 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.694361 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9d64b827-c0f3-4a50-8e75-6284cffcf625-config-volume\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:35.694648 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:35.694447 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 05 19:30:35.694648 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:35.694523 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls podName:39d68362-0254-4bd3-925d-cd2c5767a5d8 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:36.194504186 +0000 UTC m=+37.194522272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-cj69j" (UID: "39d68362-0254-4bd3-925d-cd2c5767a5d8") : secret "samples-operator-tls" not found Jul 05 19:30:35.694648 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.694585 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d64b827-c0f3-4a50-8e75-6284cffcf625-tmp-dir\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:35.694992 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.694913 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d50d6beb-5c77-4e3e-a170-a9220239ce86-trusted-ca\") pod \"console-operator-68b757fb67-qb5w5\" (UID: \"d50d6beb-5c77-4e3e-a170-a9220239ce86\") " pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:35.694992 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.694976 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/81eba20f-cd1e-41cb-a868-ee8f133a20ef-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-k8pzp\" (UID: \"81eba20f-cd1e-41cb-a868-ee8f133a20ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" Jul 05 19:30:35.695118 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:35.695000 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 19:30:35.695118 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:35.695062 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls podName:9d64b827-c0f3-4a50-8e75-6284cffcf625 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:36.19502785 +0000 UTC m=+37.195045933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls") pod "dns-default-6vhql" (UID: "9d64b827-c0f3-4a50-8e75-6284cffcf625") : secret "dns-default-metrics-tls" not found Jul 05 19:30:35.695352 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.695326 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d50d6beb-5c77-4e3e-a170-a9220239ce86-config\") pod \"console-operator-68b757fb67-qb5w5\" (UID: \"d50d6beb-5c77-4e3e-a170-a9220239ce86\") " pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:35.696815 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.696780 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d50d6beb-5c77-4e3e-a170-a9220239ce86-serving-cert\") pod \"console-operator-68b757fb67-qb5w5\" (UID: \"d50d6beb-5c77-4e3e-a170-a9220239ce86\") " pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:35.697063 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.697019 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85f8e787-c095-491d-91e6-2840a6364447-cert\") pod \"ingress-canary-4vprg\" (UID: \"85f8e787-c095-491d-91e6-2840a6364447\") " pod="openshift-ingress-canary/ingress-canary-4vprg" Jul 05 19:30:35.697303 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.697272 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1071794b-4f80-4a07-a571-e790c329a568-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-tn8sz\" (UID: \"1071794b-4f80-4a07-a571-e790c329a568\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" Jul 05 19:30:35.697701 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.697659 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/81eba20f-cd1e-41cb-a868-ee8f133a20ef-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-k8pzp\" (UID: \"81eba20f-cd1e-41cb-a868-ee8f133a20ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" Jul 05 19:30:35.704520 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.702656 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbqhp\" (UniqueName: \"kubernetes.io/projected/d50d6beb-5c77-4e3e-a170-a9220239ce86-kube-api-access-jbqhp\") pod \"console-operator-68b757fb67-qb5w5\" (UID: \"d50d6beb-5c77-4e3e-a170-a9220239ce86\") " pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:35.704520 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.703874 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnq8h\" (UniqueName: \"kubernetes.io/projected/9d64b827-c0f3-4a50-8e75-6284cffcf625-kube-api-access-bnq8h\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:35.706093 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.705171 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9qg7\" (UniqueName: \"kubernetes.io/projected/39d68362-0254-4bd3-925d-cd2c5767a5d8-kube-api-access-z9qg7\") pod \"cluster-samples-operator-56fcf86bf-cj69j\" (UID: \"39d68362-0254-4bd3-925d-cd2c5767a5d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" Jul 05 19:30:35.706234 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.706215 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" Jul 05 19:30:35.707170 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.707117 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntj5r\" (UniqueName: \"kubernetes.io/projected/81eba20f-cd1e-41cb-a868-ee8f133a20ef-kube-api-access-ntj5r\") pod \"kube-storage-version-migrator-operator-846bf668f7-k8pzp\" (UID: \"81eba20f-cd1e-41cb-a868-ee8f133a20ef\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" Jul 05 19:30:35.708013 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.707988 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlr5d\" (UniqueName: \"kubernetes.io/projected/f667ec66-87f7-4789-9288-875d41952cc4-kube-api-access-jlr5d\") pod \"network-check-source-989f8b8dc-hx424\" (UID: \"f667ec66-87f7-4789-9288-875d41952cc4\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hx424" Jul 05 19:30:35.708447 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.708426 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr9xg\" (UniqueName: \"kubernetes.io/projected/85f8e787-c095-491d-91e6-2840a6364447-kube-api-access-hr9xg\") pod \"ingress-canary-4vprg\" (UID: \"85f8e787-c095-491d-91e6-2840a6364447\") " pod="openshift-ingress-canary/ingress-canary-4vprg" Jul 05 19:30:35.722764 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.722706 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xjz8\" (UniqueName: \"kubernetes.io/projected/1071794b-4f80-4a07-a571-e790c329a568-kube-api-access-7xjz8\") pod \"service-ca-operator-5d46d9ffd4-tn8sz\" (UID: \"1071794b-4f80-4a07-a571-e790c329a568\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" Jul 05 19:30:35.749333 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.749300 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-9rf6c" Jul 05 19:30:35.757463 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.757439 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:35.771273 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.771251 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" Jul 05 19:30:35.779029 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.779010 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" Jul 05 19:30:35.785198 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.785179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hx424" Jul 05 19:30:35.799137 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:35.799112 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4vprg" Jul 05 19:30:36.098714 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:36.098668 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xcnwf\" (UID: \"90a56a02-5f5d-48bd-9285-c4db913af929\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:30:36.098909 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:36.098726 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:36.098909 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:36.098817 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:36.098909 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:36.098837 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 05 19:30:36.099069 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:36.098911 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls podName:90a56a02-5f5d-48bd-9285-c4db913af929 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:37.098889379 +0000 UTC m=+38.098907465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-xcnwf" (UID: "90a56a02-5f5d-48bd-9285-c4db913af929") : secret "cluster-monitoring-operator-tls" not found Jul 05 19:30:36.099069 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:36.098912 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 05 19:30:36.099069 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:36.098927 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 19:30:36.099069 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:36.098950 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7575585c8f-8gvjn: secret "image-registry-tls" not found Jul 05 19:30:36.099069 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:36.098967 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs podName:9d9bd347-ceb3-4b36-8654-7f2ddbecf9af nodeName:}" failed. No retries permitted until 2026-07-05 19:30:37.098951479 +0000 UTC m=+38.098969569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs") pod "router-default-9d66c8c9c-mg7q2" (UID: "9d9bd347-ceb3-4b36-8654-7f2ddbecf9af") : secret "router-metrics-certs-default" not found Jul 05 19:30:36.099069 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:36.099004 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:36.099293 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:36.099081 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle podName:9d9bd347-ceb3-4b36-8654-7f2ddbecf9af nodeName:}" failed. No retries permitted until 2026-07-05 19:30:37.099064233 +0000 UTC m=+38.099082337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle") pod "router-default-9d66c8c9c-mg7q2" (UID: "9d9bd347-ceb3-4b36-8654-7f2ddbecf9af") : configmap references non-existent config key: service-ca.crt Jul 05 19:30:36.099293 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:36.099102 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls podName:8cab1cf3-0844-475a-9a85-45b7621f7be0 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:37.099092401 +0000 UTC m=+38.099110487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls") pod "image-registry-7575585c8f-8gvjn" (UID: "8cab1cf3-0844-475a-9a85-45b7621f7be0") : secret "image-registry-tls" not found Jul 05 19:30:36.200340 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:36.200307 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:36.200340 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:36.200344 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-cj69j\" (UID: \"39d68362-0254-4bd3-925d-cd2c5767a5d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" Jul 05 19:30:36.200570 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:36.200469 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 05 19:30:36.200570 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:36.200525 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 19:30:36.200570 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:36.200530 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls podName:39d68362-0254-4bd3-925d-cd2c5767a5d8 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:37.200512852 +0000 UTC m=+38.200530939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-cj69j" (UID: "39d68362-0254-4bd3-925d-cd2c5767a5d8") : secret "samples-operator-tls" not found Jul 05 19:30:36.200570 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:36.200572 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls podName:9d64b827-c0f3-4a50-8e75-6284cffcf625 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:37.200561278 +0000 UTC m=+38.200579369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls") pod "dns-default-6vhql" (UID: "9d64b827-c0f3-4a50-8e75-6284cffcf625") : secret "dns-default-metrics-tls" not found Jul 05 19:30:36.463553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:36.463465 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:30:36.463553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:36.463517 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:30:36.463767 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:36.463478 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:30:36.467868 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:36.467851 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rvhxs\"" Jul 05 19:30:36.468004 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:36.467876 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-698zg\"" Jul 05 19:30:36.468004 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:36.467898 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 05 19:30:36.468149 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:36.468109 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 05 19:30:37.111091 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.111032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:37.111709 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.111122 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:37.111709 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:37.111174 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 05 19:30:37.111709 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.111178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xcnwf\" (UID: \"90a56a02-5f5d-48bd-9285-c4db913af929\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:30:37.111709 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:37.111252 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 05 19:30:37.111709 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:37.111260 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs podName:9d9bd347-ceb3-4b36-8654-7f2ddbecf9af nodeName:}" failed. No retries permitted until 2026-07-05 19:30:39.111239471 +0000 UTC m=+40.111257572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs") pod "router-default-9d66c8c9c-mg7q2" (UID: "9d9bd347-ceb3-4b36-8654-7f2ddbecf9af") : secret "router-metrics-certs-default" not found Jul 05 19:30:37.111709 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:37.111345 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle podName:9d9bd347-ceb3-4b36-8654-7f2ddbecf9af nodeName:}" failed. No retries permitted until 2026-07-05 19:30:39.111329372 +0000 UTC m=+40.111347462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle") pod "router-default-9d66c8c9c-mg7q2" (UID: "9d9bd347-ceb3-4b36-8654-7f2ddbecf9af") : configmap references non-existent config key: service-ca.crt Jul 05 19:30:37.111709 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:37.111364 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls podName:90a56a02-5f5d-48bd-9285-c4db913af929 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:39.111355112 +0000 UTC m=+40.111373202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-xcnwf" (UID: "90a56a02-5f5d-48bd-9285-c4db913af929") : secret "cluster-monitoring-operator-tls" not found Jul 05 19:30:37.111709 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.111406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:37.111709 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:37.111561 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 19:30:37.111709 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:37.111572 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7575585c8f-8gvjn: secret "image-registry-tls" not found Jul 05 19:30:37.111709 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:37.111605 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls podName:8cab1cf3-0844-475a-9a85-45b7621f7be0 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:39.11159458 +0000 UTC m=+40.111612675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls") pod "image-registry-7575585c8f-8gvjn" (UID: "8cab1cf3-0844-475a-9a85-45b7621f7be0") : secret "image-registry-tls" not found Jul 05 19:30:37.213638 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.213342 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:37.213795 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.213659 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-cj69j\" (UID: \"39d68362-0254-4bd3-925d-cd2c5767a5d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" Jul 05 19:30:37.214348 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:37.213885 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 05 19:30:37.214348 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:37.213990 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls podName:39d68362-0254-4bd3-925d-cd2c5767a5d8 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:39.213967292 +0000 UTC m=+40.213985399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-cj69j" (UID: "39d68362-0254-4bd3-925d-cd2c5767a5d8") : secret "samples-operator-tls" not found Jul 05 19:30:37.214348 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:37.214207 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 19:30:37.214348 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:37.214270 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls podName:9d64b827-c0f3-4a50-8e75-6284cffcf625 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:39.214252062 +0000 UTC m=+40.214270148 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls") pod "dns-default-6vhql" (UID: "9d64b827-c0f3-4a50-8e75-6284cffcf625") : secret "dns-default-metrics-tls" not found Jul 05 19:30:37.397918 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.397891 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-wz8gp"] Jul 05 19:30:37.402819 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:30:37.402793 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc38af11_062b_458d_8f23_4d34f9fd1fdd.slice/crio-a03e3bbe2143e7006a1e8587e05c99a738bc0497ff309e56824971eb80c45147 WatchSource:0}: Error finding container a03e3bbe2143e7006a1e8587e05c99a738bc0497ff309e56824971eb80c45147: Status 404 returned error can't find the container with id a03e3bbe2143e7006a1e8587e05c99a738bc0497ff309e56824971eb80c45147 Jul 05 19:30:37.406449 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.406396 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4vprg"] Jul 05 19:30:37.409100 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.409076 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-9rf6c"] Jul 05 19:30:37.411988 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.411960 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz"] Jul 05 19:30:37.415852 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.415573 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-qb5w5"] Jul 05 19:30:37.416148 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:30:37.416020 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1071794b_4f80_4a07_a571_e790c329a568.slice/crio-26d042eca8fe217fd0f18c4d33693223778eb23422144d992c4e117dbadc55d4 WatchSource:0}: Error finding container 26d042eca8fe217fd0f18c4d33693223778eb23422144d992c4e117dbadc55d4: Status 404 returned error can't find the container with id 26d042eca8fe217fd0f18c4d33693223778eb23422144d992c4e117dbadc55d4 Jul 05 19:30:37.418411 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.417882 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp"] Jul 05 19:30:37.418670 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:30:37.418567 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50d6beb_5c77_4e3e_a170_a9220239ce86.slice/crio-38af2191066845ae255eead112239d8b84593736ec283aec860de089598f1ce1 WatchSource:0}: Error finding container 38af2191066845ae255eead112239d8b84593736ec283aec860de089598f1ce1: Status 404 returned error can't find the container with id 38af2191066845ae255eead112239d8b84593736ec283aec860de089598f1ce1 Jul 05 19:30:37.421717 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.420510 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-hx424"] Jul 05 19:30:37.422527 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:30:37.422085 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81eba20f_cd1e_41cb_a868_ee8f133a20ef.slice/crio-3983fdb822fa639e6409d843b6e84d1157bf29928a4d87c9f619d2e1be9dd371 WatchSource:0}: Error finding container 3983fdb822fa639e6409d843b6e84d1157bf29928a4d87c9f619d2e1be9dd371: Status 404 returned error can't find the container with id 3983fdb822fa639e6409d843b6e84d1157bf29928a4d87c9f619d2e1be9dd371 Jul 05 19:30:37.424780 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:30:37.424677 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf667ec66_87f7_4789_9288_875d41952cc4.slice/crio-1db2c4e246eafa0bd541301c762eeb0f60ce6ecee8ffa89972f1e2336f006a24 WatchSource:0}: Error finding container 1db2c4e246eafa0bd541301c762eeb0f60ce6ecee8ffa89972f1e2336f006a24: Status 404 returned error can't find the container with id 1db2c4e246eafa0bd541301c762eeb0f60ce6ecee8ffa89972f1e2336f006a24 Jul 05 19:30:37.621785 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.621700 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" event={"ID":"1071794b-4f80-4a07-a571-e790c329a568","Type":"ContainerStarted","Data":"26d042eca8fe217fd0f18c4d33693223778eb23422144d992c4e117dbadc55d4"} Jul 05 19:30:37.622731 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.622708 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4vprg" event={"ID":"85f8e787-c095-491d-91e6-2840a6364447","Type":"ContainerStarted","Data":"27aa673b109cde8329a2e44efb437f1f882d13d89d7c1e0256244c1261ee0d17"} Jul 05 19:30:37.623731 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.623708 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" event={"ID":"bc38af11-062b-458d-8f23-4d34f9fd1fdd","Type":"ContainerStarted","Data":"a03e3bbe2143e7006a1e8587e05c99a738bc0497ff309e56824971eb80c45147"} Jul 05 19:30:37.624712 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.624686 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-9rf6c" event={"ID":"619c0eb1-3880-4758-b423-9acf269979c0","Type":"ContainerStarted","Data":"0bb668493013926099c72029917487562a271672b10c8a6c55e51a2cbfb75637"} Jul 05 19:30:37.627491 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.627471 2573 generic.go:356] "Generic (PLEG): container finished" podID="2b2a331e-da1e-4db5-ac53-84d90da88ff3" containerID="0c253327f578441b474ab4a5a304497f3f328a578aa9fbc97c65ab49933ab792" exitCode=0 Jul 05 19:30:37.627584 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.627564 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v428c" event={"ID":"2b2a331e-da1e-4db5-ac53-84d90da88ff3","Type":"ContainerDied","Data":"0c253327f578441b474ab4a5a304497f3f328a578aa9fbc97c65ab49933ab792"} Jul 05 19:30:37.628779 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.628760 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hx424" event={"ID":"f667ec66-87f7-4789-9288-875d41952cc4","Type":"ContainerStarted","Data":"1db2c4e246eafa0bd541301c762eeb0f60ce6ecee8ffa89972f1e2336f006a24"} Jul 05 19:30:37.629878 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.629856 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" event={"ID":"d50d6beb-5c77-4e3e-a170-a9220239ce86","Type":"ContainerStarted","Data":"38af2191066845ae255eead112239d8b84593736ec283aec860de089598f1ce1"} Jul 05 19:30:37.630881 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.630865 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" event={"ID":"81eba20f-cd1e-41cb-a868-ee8f133a20ef","Type":"ContainerStarted","Data":"3983fdb822fa639e6409d843b6e84d1157bf29928a4d87c9f619d2e1be9dd371"} Jul 05 19:30:37.949369 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:37.949275 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:38.030580 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:38.030540 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:38.641354 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:38.641314 2573 generic.go:356] "Generic (PLEG): container finished" podID="2b2a331e-da1e-4db5-ac53-84d90da88ff3" containerID="dd1a2c542434e8a930091dbe1e69ca25fb919bca89bf05eade90818d845690e6" exitCode=0 Jul 05 19:30:38.641789 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:38.641403 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v428c" event={"ID":"2b2a331e-da1e-4db5-ac53-84d90da88ff3","Type":"ContainerDied","Data":"dd1a2c542434e8a930091dbe1e69ca25fb919bca89bf05eade90818d845690e6"} Jul 05 19:30:39.133377 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:39.133339 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:39.133604 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:39.133457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:39.133604 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:39.133509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:39.133604 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:39.133585 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xcnwf\" (UID: \"90a56a02-5f5d-48bd-9285-c4db913af929\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:30:39.133901 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:39.133713 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 05 19:30:39.133901 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:39.133777 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls podName:90a56a02-5f5d-48bd-9285-c4db913af929 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:43.13375838 +0000 UTC m=+44.133776467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-xcnwf" (UID: "90a56a02-5f5d-48bd-9285-c4db913af929") : secret "cluster-monitoring-operator-tls" not found Jul 05 19:30:39.134509 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:39.134202 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 19:30:39.134509 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:39.134221 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7575585c8f-8gvjn: secret "image-registry-tls" not found Jul 05 19:30:39.134509 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:39.134268 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls podName:8cab1cf3-0844-475a-9a85-45b7621f7be0 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:43.134251211 +0000 UTC m=+44.134269296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls") pod "image-registry-7575585c8f-8gvjn" (UID: "8cab1cf3-0844-475a-9a85-45b7621f7be0") : secret "image-registry-tls" not found Jul 05 19:30:39.134509 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:39.134332 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 05 19:30:39.134509 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:39.134363 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs podName:9d9bd347-ceb3-4b36-8654-7f2ddbecf9af nodeName:}" failed. No retries permitted until 2026-07-05 19:30:43.134352748 +0000 UTC m=+44.134370833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs") pod "router-default-9d66c8c9c-mg7q2" (UID: "9d9bd347-ceb3-4b36-8654-7f2ddbecf9af") : secret "router-metrics-certs-default" not found Jul 05 19:30:39.134509 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:39.134424 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle podName:9d9bd347-ceb3-4b36-8654-7f2ddbecf9af nodeName:}" failed. No retries permitted until 2026-07-05 19:30:43.134414059 +0000 UTC m=+44.134432143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle") pod "router-default-9d66c8c9c-mg7q2" (UID: "9d9bd347-ceb3-4b36-8654-7f2ddbecf9af") : configmap references non-existent config key: service-ca.crt Jul 05 19:30:39.234562 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:39.234520 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:39.234728 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:39.234573 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-cj69j\" (UID: \"39d68362-0254-4bd3-925d-cd2c5767a5d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" Jul 05 19:30:39.234802 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:39.234771 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 05 19:30:39.234981 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:39.234830 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls podName:39d68362-0254-4bd3-925d-cd2c5767a5d8 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:43.234810076 +0000 UTC m=+44.234828167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-cj69j" (UID: "39d68362-0254-4bd3-925d-cd2c5767a5d8") : secret "samples-operator-tls" not found Jul 05 19:30:39.235253 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:39.235233 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 19:30:39.235334 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:39.235289 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls podName:9d64b827-c0f3-4a50-8e75-6284cffcf625 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:43.235274286 +0000 UTC m=+44.235292375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls") pod "dns-default-6vhql" (UID: "9d64b827-c0f3-4a50-8e75-6284cffcf625") : secret "dns-default-metrics-tls" not found Jul 05 19:30:39.529858 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:39.529725 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:39.578571 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:39.578536 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:39.652498 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:39.652430 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v428c" event={"ID":"2b2a331e-da1e-4db5-ac53-84d90da88ff3","Type":"ContainerStarted","Data":"4603067311e662f9e5f1c9ef672e74d764c97962fe98d335ee006d6fc723e504"} Jul 05 19:30:39.684388 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:39.683547 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v428c" podStartSLOduration=8.786840612 podStartE2EDuration="40.683527449s" totalImagesPullingTime="31.896686837s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:29:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:39.681768177 +0000 UTC m=+40.681786283" watchObservedRunningTime="2026-07-05 19:30:39.683527449 +0000 UTC m=+40.683545555" Jul 05 19:30:39.884946 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:39.884911 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:39.984448 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:39.984335 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:40.277217 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:40.277186 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:40.329655 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:40.329620 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:40.390249 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:40.390215 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:40.657025 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:40.656940 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-9rf6c" event={"ID":"619c0eb1-3880-4758-b423-9acf269979c0","Type":"ContainerStarted","Data":"2183266d7c04a162200e9e3ef350b3d6fb01c21df0c638b158b2b1616191fe31"} Jul 05 19:30:40.673638 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:40.673580 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-9rf6c" podStartSLOduration=33.471462476 podStartE2EDuration="35.673563142s" totalImagesPullingTime="2.202100666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:40.672820889 +0000 UTC m=+41.672838994" watchObservedRunningTime="2026-07-05 19:30:40.673563142 +0000 UTC m=+41.673581247" Jul 05 19:30:40.753627 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:40.753586 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:40.851967 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:40.851930 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:41.277883 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:41.277839 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:41.325696 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:41.325663 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:41.480530 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:41.480489 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:41.530827 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:41.530738 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:42.810265 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:42.810222 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:42.860568 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:42.860531 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:43.169145 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:43.169032 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xcnwf\" (UID: \"90a56a02-5f5d-48bd-9285-c4db913af929\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:30:43.169145 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:43.169093 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:43.169353 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:43.169177 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:43.169353 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:43.169216 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 05 19:30:43.169353 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:43.169252 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 19:30:43.169353 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:43.169266 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7575585c8f-8gvjn: secret "image-registry-tls" not found Jul 05 19:30:43.169353 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:43.169220 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:43.169353 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:43.169304 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls podName:90a56a02-5f5d-48bd-9285-c4db913af929 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:51.169283247 +0000 UTC m=+52.169301361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-xcnwf" (UID: "90a56a02-5f5d-48bd-9285-c4db913af929") : secret "cluster-monitoring-operator-tls" not found Jul 05 19:30:43.169353 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:43.169326 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle podName:9d9bd347-ceb3-4b36-8654-7f2ddbecf9af nodeName:}" failed. No retries permitted until 2026-07-05 19:30:51.169317839 +0000 UTC m=+52.169335922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle") pod "router-default-9d66c8c9c-mg7q2" (UID: "9d9bd347-ceb3-4b36-8654-7f2ddbecf9af") : configmap references non-existent config key: service-ca.crt Jul 05 19:30:43.169353 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:43.169344 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls podName:8cab1cf3-0844-475a-9a85-45b7621f7be0 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:51.169334909 +0000 UTC m=+52.169352994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls") pod "image-registry-7575585c8f-8gvjn" (UID: "8cab1cf3-0844-475a-9a85-45b7621f7be0") : secret "image-registry-tls" not found Jul 05 19:30:43.169353 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:43.169358 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 05 19:30:43.169785 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:43.169399 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs podName:9d9bd347-ceb3-4b36-8654-7f2ddbecf9af nodeName:}" failed. No retries permitted until 2026-07-05 19:30:51.169383097 +0000 UTC m=+52.169401183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs") pod "router-default-9d66c8c9c-mg7q2" (UID: "9d9bd347-ceb3-4b36-8654-7f2ddbecf9af") : secret "router-metrics-certs-default" not found Jul 05 19:30:43.270385 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:43.270347 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:43.270385 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:43.270392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-cj69j\" (UID: \"39d68362-0254-4bd3-925d-cd2c5767a5d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" Jul 05 19:30:43.270656 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:43.270487 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 19:30:43.270656 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:43.270502 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 05 19:30:43.270656 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:43.270546 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls podName:9d64b827-c0f3-4a50-8e75-6284cffcf625 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:51.270533478 +0000 UTC m=+52.270551562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls") pod "dns-default-6vhql" (UID: "9d64b827-c0f3-4a50-8e75-6284cffcf625") : secret "dns-default-metrics-tls" not found Jul 05 19:30:43.270656 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:43.270569 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls podName:39d68362-0254-4bd3-925d-cd2c5767a5d8 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:51.270552482 +0000 UTC m=+52.270570571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-cj69j" (UID: "39d68362-0254-4bd3-925d-cd2c5767a5d8") : secret "samples-operator-tls" not found Jul 05 19:30:45.673336 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:45.673079 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hx424" event={"ID":"f667ec66-87f7-4789-9288-875d41952cc4","Type":"ContainerStarted","Data":"b6910a38fb054e51e7a777009ff0169947f654f6e781ee79d4417c2e87bc0b91"} Jul 05 19:30:45.676029 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:45.676006 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/0.log" Jul 05 19:30:45.676187 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:45.676081 2573 generic.go:356] "Generic (PLEG): container finished" podID="d50d6beb-5c77-4e3e-a170-a9220239ce86" containerID="b28237ed03734854421e0b4b79ccec0fd5b5821088eba62157ea70b3a7f78d31" exitCode=255 Jul 05 19:30:45.676187 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:45.676152 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" event={"ID":"d50d6beb-5c77-4e3e-a170-a9220239ce86","Type":"ContainerDied","Data":"b28237ed03734854421e0b4b79ccec0fd5b5821088eba62157ea70b3a7f78d31"} Jul 05 19:30:45.676450 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:45.676425 2573 scope.go:117] "RemoveContainer" containerID="b28237ed03734854421e0b4b79ccec0fd5b5821088eba62157ea70b3a7f78d31" Jul 05 19:30:45.679016 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:45.678986 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" event={"ID":"81eba20f-cd1e-41cb-a868-ee8f133a20ef","Type":"ContainerStarted","Data":"ca8fca1ea75bca4f15d2be986c9408048cc5617f8dd89171b4ed97edfde3fba6"} Jul 05 19:30:45.681050 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:45.681014 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" event={"ID":"1071794b-4f80-4a07-a571-e790c329a568","Type":"ContainerStarted","Data":"2bf557a08904aac3bd51f5ba12f2761d53e593d80a7c896e8c0e3ad26ec40d91"} Jul 05 19:30:45.683157 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:45.683133 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4vprg" event={"ID":"85f8e787-c095-491d-91e6-2840a6364447","Type":"ContainerStarted","Data":"0921d5e3e528e53567752475b4a1ecf661896859caa409f24e76707536d7f789"} Jul 05 19:30:45.685478 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:45.685456 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" event={"ID":"bc38af11-062b-458d-8f23-4d34f9fd1fdd","Type":"ContainerStarted","Data":"5f128f4d977082d1b4145ef91a005e1380862f2b29b9b50786a9cfc25bd46840"} Jul 05 19:30:45.691141 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:45.691099 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-hx424" podStartSLOduration=27.237066676 podStartE2EDuration="34.69108382s" totalImagesPullingTime="7.454017144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:45.690613981 +0000 UTC m=+46.690632088" watchObservedRunningTime="2026-07-05 19:30:45.69108382 +0000 UTC m=+46.691101926" Jul 05 19:30:45.708098 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:45.707598 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" podStartSLOduration=37.242268881 podStartE2EDuration="44.707578758s" totalImagesPullingTime="7.465309877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:45.706874709 +0000 UTC m=+46.706892814" watchObservedRunningTime="2026-07-05 19:30:45.707578758 +0000 UTC m=+46.707596865" Jul 05 19:30:45.747388 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:45.744781 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" podStartSLOduration=28.297012364 podStartE2EDuration="35.744760713s" totalImagesPullingTime="7.447748349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:45.72553104 +0000 UTC m=+46.725549147" watchObservedRunningTime="2026-07-05 19:30:45.744760713 +0000 UTC m=+46.744778803" Jul 05 19:30:45.759221 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:45.758280 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:45.759221 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:45.758371 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:45.761491 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:45.760572 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" podStartSLOduration=27.306967899 podStartE2EDuration="34.760552614s" totalImagesPullingTime="7.453584715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:45.759958003 +0000 UTC m=+46.759976101" watchObservedRunningTime="2026-07-05 19:30:45.760552614 +0000 UTC m=+46.760570721" Jul 05 19:30:46.070414 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.070353 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4vprg" podStartSLOduration=3.61572717 podStartE2EDuration="11.070330659s" totalImagesPullingTime="7.454603489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:30:45.777340047 +0000 UTC m=+46.777358155" watchObservedRunningTime="2026-07-05 19:30:46.070330659 +0000 UTC m=+47.070348768" Jul 05 19:30:46.070858 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.070832 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8f8zn"] Jul 05 19:30:46.089395 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.089339 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:46.094278 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.094248 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8f8zn"] Jul 05 19:30:46.094414 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.094385 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.097166 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.097142 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 05 19:30:46.097345 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.097225 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 05 19:30:46.097546 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.097527 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rl664\"" Jul 05 19:30:46.196667 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.196595 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.196863 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.196760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68db1a0c-9074-4478-83cf-e9a5b517b553-crio-socket\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.196863 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.196788 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68db1a0c-9074-4478-83cf-e9a5b517b553-data-volume\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.196863 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.196809 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.197020 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.196860 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8z74\" (UniqueName: \"kubernetes.io/projected/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-api-access-b8z74\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.298195 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.298155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68db1a0c-9074-4478-83cf-e9a5b517b553-crio-socket\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.298195 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.298195 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68db1a0c-9074-4478-83cf-e9a5b517b553-data-volume\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.298446 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.298212 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.298446 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.298229 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8z74\" (UniqueName: \"kubernetes.io/projected/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-api-access-b8z74\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.298446 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.298376 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/68db1a0c-9074-4478-83cf-e9a5b517b553-crio-socket\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.298593 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:46.298385 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 05 19:30:46.298593 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:46.298547 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm podName:68db1a0c-9074-4478-83cf-e9a5b517b553 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:46.798524984 +0000 UTC m=+47.798543074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8f8zn" (UID: "68db1a0c-9074-4478-83cf-e9a5b517b553") : configmap "kube-rbac-proxy" not found Jul 05 19:30:46.298593 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:46.298486 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 19:30:46.298593 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.298581 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/68db1a0c-9074-4478-83cf-e9a5b517b553-data-volume\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.298784 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:46.298608 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls podName:68db1a0c-9074-4478-83cf-e9a5b517b553 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:46.798590302 +0000 UTC m=+47.798608411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8f8zn" (UID: "68db1a0c-9074-4478-83cf-e9a5b517b553") : secret "insights-runtime-extractor-tls" not found Jul 05 19:30:46.298784 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.298389 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.309790 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.309640 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8z74\" (UniqueName: \"kubernetes.io/projected/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-api-access-b8z74\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.692125 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.692097 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 19:30:46.692579 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.692479 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/0.log" Jul 05 19:30:46.692579 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.692510 2573 generic.go:356] "Generic (PLEG): container finished" podID="d50d6beb-5c77-4e3e-a170-a9220239ce86" containerID="56f472e923acae8ae88b01c4db934eaaa0af914cf0388e126d9ec0a2dca15fe2" exitCode=255 Jul 05 19:30:46.692579 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.692538 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" event={"ID":"d50d6beb-5c77-4e3e-a170-a9220239ce86","Type":"ContainerDied","Data":"56f472e923acae8ae88b01c4db934eaaa0af914cf0388e126d9ec0a2dca15fe2"} Jul 05 19:30:46.692728 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.692618 2573 scope.go:117] "RemoveContainer" containerID="b28237ed03734854421e0b4b79ccec0fd5b5821088eba62157ea70b3a7f78d31" Jul 05 19:30:46.692860 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.692841 2573 scope.go:117] "RemoveContainer" containerID="56f472e923acae8ae88b01c4db934eaaa0af914cf0388e126d9ec0a2dca15fe2" Jul 05 19:30:46.693211 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:46.693186 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-qb5w5_openshift-console-operator(d50d6beb-5c77-4e3e-a170-a9220239ce86)\"" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" podUID="d50d6beb-5c77-4e3e-a170-a9220239ce86" Jul 05 19:30:46.766586 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.766553 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:46.802697 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.802666 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.802877 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:46.802808 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 05 19:30:46.802942 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:46.802915 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm podName:68db1a0c-9074-4478-83cf-e9a5b517b553 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:47.80289264 +0000 UTC m=+48.802910725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8f8zn" (UID: "68db1a0c-9074-4478-83cf-e9a5b517b553") : configmap "kube-rbac-proxy" not found Jul 05 19:30:46.803347 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.803138 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:46.803347 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:46.803319 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 19:30:46.803756 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:46.803519 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls podName:68db1a0c-9074-4478-83cf-e9a5b517b553 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:47.803500417 +0000 UTC m=+48.803518508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8f8zn" (UID: "68db1a0c-9074-4478-83cf-e9a5b517b553") : secret "insights-runtime-extractor-tls" not found Jul 05 19:30:46.866996 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.866962 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:46.919265 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.919217 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:46.988445 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:46.988361 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:47.043888 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:47.043850 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:47.341231 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:47.341200 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:47.441470 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:47.441436 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:47.697679 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:47.697600 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 19:30:47.698150 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:47.698034 2573 scope.go:117] "RemoveContainer" containerID="56f472e923acae8ae88b01c4db934eaaa0af914cf0388e126d9ec0a2dca15fe2" Jul 05 19:30:47.698291 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:47.698269 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-qb5w5_openshift-console-operator(d50d6beb-5c77-4e3e-a170-a9220239ce86)\"" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" podUID="d50d6beb-5c77-4e3e-a170-a9220239ce86" Jul 05 19:30:47.771337 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:47.771299 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:47.810568 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:47.810535 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:47.810713 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:47.810692 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 19:30:47.810751 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:47.810716 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:47.810786 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:47.810762 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls podName:68db1a0c-9074-4478-83cf-e9a5b517b553 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:49.810743158 +0000 UTC m=+50.810761265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8f8zn" (UID: "68db1a0c-9074-4478-83cf-e9a5b517b553") : secret "insights-runtime-extractor-tls" not found Jul 05 19:30:47.810831 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:47.810788 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 05 19:30:47.810831 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:47.810816 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm podName:68db1a0c-9074-4478-83cf-e9a5b517b553 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:49.810809481 +0000 UTC m=+50.810827565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8f8zn" (UID: "68db1a0c-9074-4478-83cf-e9a5b517b553") : configmap "kube-rbac-proxy" not found Jul 05 19:30:48.050825 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:48.050787 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:48.108976 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:48.108938 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:48.231579 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:48.231550 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xnmxp_5c96e505-de6a-4853-822c-e80de519e002/dns-node-resolver/0.log" Jul 05 19:30:48.832183 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:48.832152 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4snds_cb2e6493-927f-48b2-8e47-16dc2379911a/node-ca/0.log" Jul 05 19:30:49.634227 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:49.634194 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4vprg_85f8e787-c095-491d-91e6-2840a6364447/serve-healthcheck-canary/0.log" Jul 05 19:30:49.676831 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:49.676795 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:49.737234 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:49.737199 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:49.828488 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:49.828451 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:49.828656 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:49.828512 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:49.828656 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:49.828584 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 05 19:30:49.828656 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:49.828592 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 19:30:49.828656 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:49.828647 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm podName:68db1a0c-9074-4478-83cf-e9a5b517b553 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:53.828633819 +0000 UTC m=+54.828651903 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8f8zn" (UID: "68db1a0c-9074-4478-83cf-e9a5b517b553") : configmap "kube-rbac-proxy" not found Jul 05 19:30:49.828790 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:49.828662 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls podName:68db1a0c-9074-4478-83cf-e9a5b517b553 nodeName:}" failed. No retries permitted until 2026-07-05 19:30:53.828655096 +0000 UTC m=+54.828673180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8f8zn" (UID: "68db1a0c-9074-4478-83cf-e9a5b517b553") : secret "insights-runtime-extractor-tls" not found Jul 05 19:30:50.515507 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:50.515468 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:51.241797 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:51.241760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xcnwf\" (UID: \"90a56a02-5f5d-48bd-9285-c4db913af929\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:30:51.241797 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:51.241798 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:30:51.242061 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:51.241907 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 19:30:51.242061 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:51.241920 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7575585c8f-8gvjn: secret "image-registry-tls" not found Jul 05 19:30:51.242061 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:51.241918 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 05 19:30:51.242061 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:51.241970 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls podName:8cab1cf3-0844-475a-9a85-45b7621f7be0 nodeName:}" failed. No retries permitted until 2026-07-05 19:31:07.241956345 +0000 UTC m=+68.241974429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls") pod "image-registry-7575585c8f-8gvjn" (UID: "8cab1cf3-0844-475a-9a85-45b7621f7be0") : secret "image-registry-tls" not found Jul 05 19:30:51.242061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:51.241992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:51.242061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:51.242025 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:30:51.242305 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:51.242110 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 05 19:30:51.242305 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:51.242116 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls podName:90a56a02-5f5d-48bd-9285-c4db913af929 nodeName:}" failed. No retries permitted until 2026-07-05 19:31:07.242027499 +0000 UTC m=+68.242045599 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-xcnwf" (UID: "90a56a02-5f5d-48bd-9285-c4db913af929") : secret "cluster-monitoring-operator-tls" not found Jul 05 19:30:51.242305 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:51.242159 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs podName:9d9bd347-ceb3-4b36-8654-7f2ddbecf9af nodeName:}" failed. No retries permitted until 2026-07-05 19:31:07.242149042 +0000 UTC m=+68.242167132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs") pod "router-default-9d66c8c9c-mg7q2" (UID: "9d9bd347-ceb3-4b36-8654-7f2ddbecf9af") : secret "router-metrics-certs-default" not found Jul 05 19:30:51.242305 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:51.242200 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle podName:9d9bd347-ceb3-4b36-8654-7f2ddbecf9af nodeName:}" failed. No retries permitted until 2026-07-05 19:31:07.242182581 +0000 UTC m=+68.242200665 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle") pod "router-default-9d66c8c9c-mg7q2" (UID: "9d9bd347-ceb3-4b36-8654-7f2ddbecf9af") : configmap references non-existent config key: service-ca.crt Jul 05 19:30:51.343214 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:51.343177 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:30:51.343409 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:51.343227 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-cj69j\" (UID: \"39d68362-0254-4bd3-925d-cd2c5767a5d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" Jul 05 19:30:51.343409 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:51.343387 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 05 19:30:51.343409 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:51.343391 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 19:30:51.343557 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:51.343491 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls podName:39d68362-0254-4bd3-925d-cd2c5767a5d8 nodeName:}" failed. No retries permitted until 2026-07-05 19:31:07.343477596 +0000 UTC m=+68.343495680 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-cj69j" (UID: "39d68362-0254-4bd3-925d-cd2c5767a5d8") : secret "samples-operator-tls" not found Jul 05 19:30:51.343557 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:51.343504 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls podName:9d64b827-c0f3-4a50-8e75-6284cffcf625 nodeName:}" failed. No retries permitted until 2026-07-05 19:31:07.343497672 +0000 UTC m=+68.343515756 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls") pod "dns-default-6vhql" (UID: "9d64b827-c0f3-4a50-8e75-6284cffcf625") : secret "dns-default-metrics-tls" not found Jul 05 19:30:53.181101 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:53.181021 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:53.866623 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:53.866584 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:53.866781 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:53.866648 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:30:53.866781 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:53.866747 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 19:30:53.866876 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:53.866785 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 05 19:30:53.866876 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:53.866822 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls podName:68db1a0c-9074-4478-83cf-e9a5b517b553 nodeName:}" failed. No retries permitted until 2026-07-05 19:31:01.86680672 +0000 UTC m=+62.866824805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8f8zn" (UID: "68db1a0c-9074-4478-83cf-e9a5b517b553") : secret "insights-runtime-extractor-tls" not found Jul 05 19:30:53.866876 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:53.866840 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm podName:68db1a0c-9074-4478-83cf-e9a5b517b553 nodeName:}" failed. No retries permitted until 2026-07-05 19:31:01.866827784 +0000 UTC m=+62.866845868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8f8zn" (UID: "68db1a0c-9074-4478-83cf-e9a5b517b553") : configmap "kube-rbac-proxy" not found Jul 05 19:30:53.875694 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:53.875667 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:53.973244 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:53.973213 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:54.082545 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:54.082505 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:54.140675 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:54.140586 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:54.215722 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:54.215688 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:54.971522 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:54.971490 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:55.369477 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:55.369439 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:55.758260 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:55.758232 2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:55.758428 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:55.758286 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:30:55.758669 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:55.758657 2573 scope.go:117] "RemoveContainer" containerID="56f472e923acae8ae88b01c4db934eaaa0af914cf0388e126d9ec0a2dca15fe2" Jul 05 19:30:55.758814 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:30:55.758796 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-qb5w5_openshift-console-operator(d50d6beb-5c77-4e3e-a170-a9220239ce86)\"" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" podUID="d50d6beb-5c77-4e3e-a170-a9220239ce86" Jul 05 19:30:55.778278 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:55.778242 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:56.171758 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:56.171574 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:56.373478 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:56.373415 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:56.455590 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:56.455493 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:56.583033 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:56.583000 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:56.971567 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:56.971535 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:57.480248 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:57.480220 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:57.581769 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:57.581738 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:57.682872 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:57.682831 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:57.782876 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:57.782844 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:57.970996 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:57.970958 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:58.171978 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:58.171876 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:58.272628 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:58.272594 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:58.776957 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:58.776923 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:59.289828 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:59.289797 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:59.390724 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:59.390653 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:59.675734 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:59.675648 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:59.776942 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:59.776910 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:30:59.971507 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:30:59.971425 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:00.374825 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:00.374789 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:00.611940 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:00.611914 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s4jh7" Jul 05 19:31:01.176173 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:01.176136 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:01.572878 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:01.572842 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:01.773855 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:01.773813 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:01.937904 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:01.937812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:31:01.938111 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:01.937902 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:31:01.938111 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:31:01.937966 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 19:31:01.938111 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:31:01.938070 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls podName:68db1a0c-9074-4478-83cf-e9a5b517b553 nodeName:}" failed. No retries permitted until 2026-07-05 19:31:17.938054333 +0000 UTC m=+78.938072430 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8f8zn" (UID: "68db1a0c-9074-4478-83cf-e9a5b517b553") : secret "insights-runtime-extractor-tls" not found Jul 05 19:31:01.938434 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:01.938416 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/68db1a0c-9074-4478-83cf-e9a5b517b553-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:31:01.974189 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:01.974147 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:02.174892 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:02.174839 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:03.171615 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:03.171547 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:03.370441 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:03.370405 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:03.571954 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:03.571914 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:03.972406 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:03.972312 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:04.372253 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:04.372219 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:04.583626 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:04.583580 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:05.270188 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.270144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:31:05.273179 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.273156 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 05 19:31:05.283908 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.283878 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d84a2165-5809-40c0-a5b3-7d3a976291ee-original-pull-secret\") pod \"global-pull-secret-syncer-pcsv4\" (UID: \"d84a2165-5809-40c0-a5b3-7d3a976291ee\") " pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:31:05.371211 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.371162 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs\") pod \"network-metrics-daemon-86bxp\" (UID: \"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3\") " pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:31:05.371418 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.371309 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j8b\" (UniqueName: \"kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b\") pod \"network-check-target-qsk8d\" (UID: \"de14a84c-5bb0-4f31-9894-c71098e82aec\") " pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:31:05.373839 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.373817 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 05 19:31:05.374229 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.374098 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4j8b\" (UniqueName: \"kubernetes.io/projected/de14a84c-5bb0-4f31-9894-c71098e82aec-kube-api-access-h4j8b\") pod \"network-check-target-qsk8d\" (UID: \"de14a84c-5bb0-4f31-9894-c71098e82aec\") " pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:31:05.384261 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.384236 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed74973b-94ab-43a3-a0d3-b5e89daf5cf3-metrics-certs\") pod \"network-metrics-daemon-86bxp\" (UID: \"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3\") " pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:31:05.575575 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.575485 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcsv4" Jul 05 19:31:05.585288 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.585272 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rvhxs\"" Jul 05 19:31:05.589670 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.589645 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-698zg\"" Jul 05 19:31:05.592803 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.592778 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-86bxp" Jul 05 19:31:05.597737 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.597651 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:31:05.722471 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:31:05.722428 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84a2165_5809_40c0_a5b3_7d3a976291ee.slice/crio-5f0477ccb79678353c65f536f76bbdb952a4039e7a532238585d75e36371b652 WatchSource:0}: Error finding container 5f0477ccb79678353c65f536f76bbdb952a4039e7a532238585d75e36371b652: Status 404 returned error can't find the container with id 5f0477ccb79678353c65f536f76bbdb952a4039e7a532238585d75e36371b652 Jul 05 19:31:05.722622 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.722593 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pcsv4"] Jul 05 19:31:05.739287 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.739260 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qsk8d"] Jul 05 19:31:05.742569 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:31:05.742545 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde14a84c_5bb0_4f31_9894_c71098e82aec.slice/crio-0a6ac83a165a5b64877ae635d9a2df82e9136834055e0b4b39a096a7f9172b9d WatchSource:0}: Error finding container 0a6ac83a165a5b64877ae635d9a2df82e9136834055e0b4b39a096a7f9172b9d: Status 404 returned error can't find the container with id 0a6ac83a165a5b64877ae635d9a2df82e9136834055e0b4b39a096a7f9172b9d Jul 05 19:31:05.755536 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.755503 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pcsv4" event={"ID":"d84a2165-5809-40c0-a5b3-7d3a976291ee","Type":"ContainerStarted","Data":"5f0477ccb79678353c65f536f76bbdb952a4039e7a532238585d75e36371b652"} Jul 05 19:31:05.760079 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.758005 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-86bxp"] Jul 05 19:31:05.761419 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:05.761391 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qsk8d" event={"ID":"de14a84c-5bb0-4f31-9894-c71098e82aec","Type":"ContainerStarted","Data":"0a6ac83a165a5b64877ae635d9a2df82e9136834055e0b4b39a096a7f9172b9d"} Jul 05 19:31:05.763099 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:31:05.763071 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded74973b_94ab_43a3_a0d3_b5e89daf5cf3.slice/crio-d2bf6c54476098c9a8aafc09cafc12bab46f34b1b7fd57fee0b522b15ac1eb23 WatchSource:0}: Error finding container d2bf6c54476098c9a8aafc09cafc12bab46f34b1b7fd57fee0b522b15ac1eb23: Status 404 returned error can't find the container with id d2bf6c54476098c9a8aafc09cafc12bab46f34b1b7fd57fee0b522b15ac1eb23 Jul 05 19:31:06.766737 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:06.766698 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-86bxp" event={"ID":"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3","Type":"ContainerStarted","Data":"d2bf6c54476098c9a8aafc09cafc12bab46f34b1b7fd57fee0b522b15ac1eb23"} Jul 05 19:31:06.768547 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:06.768507 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qsk8d" event={"ID":"de14a84c-5bb0-4f31-9894-c71098e82aec","Type":"ContainerStarted","Data":"88c32c4cf92cd2f13ec9b3dc91c972cd0b212942fa7b88ecfdba939638630929"} Jul 05 19:31:06.768687 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:06.768658 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:31:06.786035 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:06.785962 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qsk8d" podStartSLOduration=67.785941251 podStartE2EDuration="1m7.785941251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:06.785160792 +0000 UTC m=+67.785178900" watchObservedRunningTime="2026-07-05 19:31:06.785941251 +0000 UTC m=+67.785959359" Jul 05 19:31:07.287342 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.287297 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:31:07.287529 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.287362 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:31:07.287529 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.287441 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xcnwf\" (UID: \"90a56a02-5f5d-48bd-9285-c4db913af929\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:31:07.287529 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.287475 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:31:07.288267 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.288219 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-service-ca-bundle\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:31:07.290440 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.290414 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d9bd347-ceb3-4b36-8654-7f2ddbecf9af-metrics-certs\") pod \"router-default-9d66c8c9c-mg7q2\" (UID: \"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af\") " pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:31:07.290653 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.290625 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls\") pod \"image-registry-7575585c8f-8gvjn\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:31:07.290757 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.290625 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/90a56a02-5f5d-48bd-9285-c4db913af929-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-xcnwf\" (UID: \"90a56a02-5f5d-48bd-9285-c4db913af929\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:31:07.388711 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.388613 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:31:07.388711 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.388663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-cj69j\" (UID: \"39d68362-0254-4bd3-925d-cd2c5767a5d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" Jul 05 19:31:07.391287 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.391260 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d64b827-c0f3-4a50-8e75-6284cffcf625-metrics-tls\") pod \"dns-default-6vhql\" (UID: \"9d64b827-c0f3-4a50-8e75-6284cffcf625\") " pod="openshift-dns/dns-default-6vhql" Jul 05 19:31:07.391666 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.391641 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/39d68362-0254-4bd3-925d-cd2c5767a5d8-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-cj69j\" (UID: \"39d68362-0254-4bd3-925d-cd2c5767a5d8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" Jul 05 19:31:07.521580 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.521549 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-trq5g\"" Jul 05 19:31:07.529821 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.529796 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" Jul 05 19:31:07.533403 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.533375 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k29fx\"" Jul 05 19:31:07.541649 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.541611 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:31:07.544879 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.544630 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cghrd\"" Jul 05 19:31:07.552174 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.552150 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:31:07.566357 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.566317 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-krqrz\"" Jul 05 19:31:07.574016 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.573992 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" Jul 05 19:31:07.596612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.596369 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nkl9h\"" Jul 05 19:31:07.604904 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.603772 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6vhql" Jul 05 19:31:07.742219 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.742019 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf"] Jul 05 19:31:07.764571 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.764493 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7575585c8f-8gvjn"] Jul 05 19:31:07.785492 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.785455 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" event={"ID":"90a56a02-5f5d-48bd-9285-c4db913af929","Type":"ContainerStarted","Data":"640d687f9021c04c3c044db39962ba01d893b2dff3dcca10bb43fe17c3bae92e"} Jul 05 19:31:07.788132 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.787845 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-86bxp" event={"ID":"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3","Type":"ContainerStarted","Data":"7128a974b92315ae84f8b31f43fe153a1d4932bdea2e827bba269eee9a56796e"} Jul 05 19:31:07.788132 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.787903 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-86bxp" event={"ID":"ed74973b-94ab-43a3-a0d3-b5e89daf5cf3","Type":"ContainerStarted","Data":"8b310b9a0bfb17921ff873aeec4d0d97c1a9824b87f809e6f47ead3dcd9d7598"} Jul 05 19:31:07.797114 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.797082 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-9d66c8c9c-mg7q2"] Jul 05 19:31:07.801859 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:31:07.801828 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9bd347_ceb3_4b36_8654_7f2ddbecf9af.slice/crio-cc92483b132640ace26ab15c8d06b02420c4197030a215c6527ddcb3a619dbf4 WatchSource:0}: Error finding container cc92483b132640ace26ab15c8d06b02420c4197030a215c6527ddcb3a619dbf4: Status 404 returned error can't find the container with id cc92483b132640ace26ab15c8d06b02420c4197030a215c6527ddcb3a619dbf4 Jul 05 19:31:07.808705 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.808514 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-86bxp" podStartSLOduration=67.677042527 podStartE2EDuration="1m8.808494019s" totalImagesPullingTime="1.131451492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:07.807222809 +0000 UTC m=+68.807240915" watchObservedRunningTime="2026-07-05 19:31:07.808494019 +0000 UTC m=+68.808512125" Jul 05 19:31:07.817611 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.817573 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6vhql"] Jul 05 19:31:07.817715 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:07.817632 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j"] Jul 05 19:31:07.819697 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:31:07.819664 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d64b827_c0f3_4a50_8e75_6284cffcf625.slice/crio-9f472137144a9fb6946406bf6ed5f958387373c0f0dde6ffcd538ac619d42a29 WatchSource:0}: Error finding container 9f472137144a9fb6946406bf6ed5f958387373c0f0dde6ffcd538ac619d42a29: Status 404 returned error can't find the container with id 9f472137144a9fb6946406bf6ed5f958387373c0f0dde6ffcd538ac619d42a29 Jul 05 19:31:08.803355 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:08.802713 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" event={"ID":"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af","Type":"ContainerStarted","Data":"adf47e9c8388fb35b9b90ecfc73dcc07e5cab8b705021ebf453ab23c7ff6d2e3"} Jul 05 19:31:08.803355 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:08.802860 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" event={"ID":"9d9bd347-ceb3-4b36-8654-7f2ddbecf9af","Type":"ContainerStarted","Data":"cc92483b132640ace26ab15c8d06b02420c4197030a215c6527ddcb3a619dbf4"} Jul 05 19:31:08.806691 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:08.806608 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vhql" event={"ID":"9d64b827-c0f3-4a50-8e75-6284cffcf625","Type":"ContainerStarted","Data":"9f472137144a9fb6946406bf6ed5f958387373c0f0dde6ffcd538ac619d42a29"} Jul 05 19:31:08.808578 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:08.808513 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" event={"ID":"39d68362-0254-4bd3-925d-cd2c5767a5d8","Type":"ContainerStarted","Data":"dca86cd7b0063402e19bb32963222415a679dad9016b2e0e5aa414abc8ca20e8"} Jul 05 19:31:08.811406 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:08.811340 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" event={"ID":"8cab1cf3-0844-475a-9a85-45b7621f7be0","Type":"ContainerStarted","Data":"e5be2000613131b780fa4be6b25f1dd4e6d9016b264a7eb50d91bed17ff55730"} Jul 05 19:31:08.811406 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:08.811375 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" event={"ID":"8cab1cf3-0844-475a-9a85-45b7621f7be0","Type":"ContainerStarted","Data":"0c778c7db50d9bd99ea22fab12ae92cf276cd6150f3e57a24a9ecbe81265aadf"} Jul 05 19:31:08.811609 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:08.811583 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:31:08.845739 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:08.844384 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" podStartSLOduration=68.844365351 podStartE2EDuration="1m8.844365351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:08.84398028 +0000 UTC m=+69.843998386" watchObservedRunningTime="2026-07-05 19:31:08.844365351 +0000 UTC m=+69.844383459" Jul 05 19:31:08.845739 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:08.845596 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" podStartSLOduration=67.845578948 podStartE2EDuration="1m7.845578948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:08.825219579 +0000 UTC m=+69.825237690" watchObservedRunningTime="2026-07-05 19:31:08.845578948 +0000 UTC m=+69.845597055" Jul 05 19:31:09.466451 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:09.465548 2573 scope.go:117] "RemoveContainer" containerID="56f472e923acae8ae88b01c4db934eaaa0af914cf0388e126d9ec0a2dca15fe2" Jul 05 19:31:09.552682 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:09.552643 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:31:09.555748 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:09.555722 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:31:09.815116 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:09.815076 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:31:09.816291 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:09.816267 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-9d66c8c9c-mg7q2" Jul 05 19:31:11.224553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:11.224502 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:11.299123 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:11.299087 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:11.403426 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:11.403319 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:11.498983 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:11.498953 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:11.599336 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:11.599302 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:11.660921 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:11.660841 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:12.596106 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:12.596068 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:12.697203 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:12.697074 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:12.834282 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:12.834193 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" event={"ID":"39d68362-0254-4bd3-925d-cd2c5767a5d8","Type":"ContainerStarted","Data":"d5eb405f945daafe260ae0bde48415f45a99f4ee14a02cf4966f76d42f65fb6f"} Jul 05 19:31:12.840794 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:12.840657 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 19:31:12.840794 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:12.840719 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" event={"ID":"d50d6beb-5c77-4e3e-a170-a9220239ce86","Type":"ContainerStarted","Data":"a7d6a7d23582796fdb7269f30f415a9f2a41908b7959bcacfefa1b5c6f5a95c8"} Jul 05 19:31:12.845186 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:12.845162 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:31:13.112034 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.112000 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:13.213185 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.213097 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:13.275367 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.275322 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:13.494711 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.494619 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:13.844711 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.844648 2573 patch_prober.go:28] interesting pod/console-operator-68b757fb67-qb5w5 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.11:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 05 19:31:13.845117 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.844758 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" podUID="d50d6beb-5c77-4e3e-a170-a9220239ce86" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 05 19:31:13.847636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.847595 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" event={"ID":"90a56a02-5f5d-48bd-9285-c4db913af929","Type":"ContainerStarted","Data":"88522188fed6a74fc539be9d20fee246513947edecf8b6b9643678f6f6336212"} Jul 05 19:31:13.849974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.849542 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pcsv4" event={"ID":"d84a2165-5809-40c0-a5b3-7d3a976291ee","Type":"ContainerStarted","Data":"57b15e0243185a515489ec2ee4b6bf35248f66b027e2a847289c62b6b92ac918"} Jul 05 19:31:13.856617 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.856588 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vhql" event={"ID":"9d64b827-c0f3-4a50-8e75-6284cffcf625","Type":"ContainerStarted","Data":"c20ff61aa46fc4ce068346e7b7800e186e7f549772c198b3577d868e1a8c446f"} Jul 05 19:31:13.856761 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.856677 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vhql" event={"ID":"9d64b827-c0f3-4a50-8e75-6284cffcf625","Type":"ContainerStarted","Data":"3992f41f0d4eeeea0571887cce3976c3b7055100e3a4ce24843d3ca1b2e7a0a4"} Jul 05 19:31:13.856761 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.856709 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6vhql" Jul 05 19:31:13.858245 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.858218 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" event={"ID":"39d68362-0254-4bd3-925d-cd2c5767a5d8","Type":"ContainerStarted","Data":"c535e87e837eeb2e0670ae6c77a7e80d98875fc32d459182ca8315d1959833dc"} Jul 05 19:31:13.862085 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.862062 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" Jul 05 19:31:13.868437 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.868396 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-xcnwf" podStartSLOduration=67.958565213 podStartE2EDuration="1m12.868383044s" totalImagesPullingTime="4.909817831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:13.86771355 +0000 UTC m=+74.867731657" watchObservedRunningTime="2026-07-05 19:31:13.868383044 +0000 UTC m=+74.868401149" Jul 05 19:31:13.868662 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.868643 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-qb5w5" podStartSLOduration=60.419373224 podStartE2EDuration="1m7.868637164s" totalImagesPullingTime="7.44926394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:12.863716921 +0000 UTC m=+73.863735028" watchObservedRunningTime="2026-07-05 19:31:13.868637164 +0000 UTC m=+74.868655269" Jul 05 19:31:13.897150 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.897086 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6vhql" podStartSLOduration=34.05471386 podStartE2EDuration="38.897070061s" totalImagesPullingTime="4.842356201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:13.895977955 +0000 UTC m=+74.895996074" watchObservedRunningTime="2026-07-05 19:31:13.897070061 +0000 UTC m=+74.897088166" Jul 05 19:31:13.926989 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.926918 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pcsv4" podStartSLOduration=67.989549999 podStartE2EDuration="1m14.926895688s" totalImagesPullingTime="6.937345689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:29:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:13.925151085 +0000 UTC m=+74.925169192" watchObservedRunningTime="2026-07-05 19:31:13.926895688 +0000 UTC m=+74.926913795" Jul 05 19:31:13.940792 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:13.940736 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-cj69j" podStartSLOduration=64.153786948 podStartE2EDuration="1m8.94071919s" totalImagesPullingTime="4.786932242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:13.939948856 +0000 UTC m=+74.939966962" watchObservedRunningTime="2026-07-05 19:31:13.94071919 +0000 UTC m=+74.940737296" Jul 05 19:31:14.032558 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.032526 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-p6dfl"] Jul 05 19:31:14.053205 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.053158 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:14.056493 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.056465 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-p6dfl"] Jul 05 19:31:14.056598 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.056568 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-p6dfl" Jul 05 19:31:14.059390 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.059368 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 05 19:31:14.059390 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.059382 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 05 19:31:14.059583 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.059451 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-df5bl\"" Jul 05 19:31:14.111379 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.111281 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:14.160147 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.160114 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm4pj\" (UniqueName: \"kubernetes.io/projected/fa57f98a-83b8-4dfb-8c4e-97126b6b9cbe-kube-api-access-zm4pj\") pod \"downloads-685bfb5f64-p6dfl\" (UID: \"fa57f98a-83b8-4dfb-8c4e-97126b6b9cbe\") " pod="openshift-console/downloads-685bfb5f64-p6dfl" Jul 05 19:31:14.175690 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.175661 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:14.260920 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.260885 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm4pj\" (UniqueName: \"kubernetes.io/projected/fa57f98a-83b8-4dfb-8c4e-97126b6b9cbe-kube-api-access-zm4pj\") pod \"downloads-685bfb5f64-p6dfl\" (UID: \"fa57f98a-83b8-4dfb-8c4e-97126b6b9cbe\") " pod="openshift-console/downloads-685bfb5f64-p6dfl" Jul 05 19:31:14.270802 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.270776 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm4pj\" (UniqueName: \"kubernetes.io/projected/fa57f98a-83b8-4dfb-8c4e-97126b6b9cbe-kube-api-access-zm4pj\") pod \"downloads-685bfb5f64-p6dfl\" (UID: \"fa57f98a-83b8-4dfb-8c4e-97126b6b9cbe\") " pod="openshift-console/downloads-685bfb5f64-p6dfl" Jul 05 19:31:14.342181 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.342149 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:14.365980 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.365902 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-p6dfl" Jul 05 19:31:14.443593 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.443559 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:14.513245 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.513214 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-p6dfl"] Jul 05 19:31:14.516275 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.516251 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:14.516498 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:31:14.516477 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa57f98a_83b8_4dfb_8c4e_97126b6b9cbe.slice/crio-c1bbd1b00c47bcd460abc04354e02a53154c4be1d2cf7deac6057615725279e5 WatchSource:0}: Error finding container c1bbd1b00c47bcd460abc04354e02a53154c4be1d2cf7deac6057615725279e5: Status 404 returned error can't find the container with id c1bbd1b00c47bcd460abc04354e02a53154c4be1d2cf7deac6057615725279e5 Jul 05 19:31:14.707949 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.707860 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:14.761735 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.761697 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:14.863552 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:14.863513 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-p6dfl" event={"ID":"fa57f98a-83b8-4dfb-8c4e-97126b6b9cbe","Type":"ContainerStarted","Data":"c1bbd1b00c47bcd460abc04354e02a53154c4be1d2cf7deac6057615725279e5"} Jul 05 19:31:15.108056 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:15.108007 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:15.207944 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:15.207903 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:15.335525 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:15.335464 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:15.436136 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:15.436028 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:15.722630 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:15.722547 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:15.923622 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:15.923586 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:16.123557 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:16.123524 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:16.174101 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:16.174067 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:17.738441 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:17.738397 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:17.841961 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:17.841916 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:17.989998 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:17.989913 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:31:17.992875 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:17.992841 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/68db1a0c-9074-4478-83cf-e9a5b517b553-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8f8zn\" (UID: \"68db1a0c-9074-4478-83cf-e9a5b517b553\") " pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:31:18.209717 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:18.209686 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rl664\"" Jul 05 19:31:18.217999 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:18.217965 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8f8zn" Jul 05 19:31:18.297006 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:18.296972 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:18.384195 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:18.384167 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8f8zn"] Jul 05 19:31:18.385989 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:18.385965 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:18.386876 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:31:18.386545 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68db1a0c_9074_4478_83cf_e9a5b517b553.slice/crio-15dc8f7c45f09365012549c864d0d286641c1c6ea28253d2b9feed2fa75a49c1 WatchSource:0}: Error finding container 15dc8f7c45f09365012549c864d0d286641c1c6ea28253d2b9feed2fa75a49c1: Status 404 returned error can't find the container with id 15dc8f7c45f09365012549c864d0d286641c1c6ea28253d2b9feed2fa75a49c1 Jul 05 19:31:18.882389 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:18.882328 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8f8zn" event={"ID":"68db1a0c-9074-4478-83cf-e9a5b517b553","Type":"ContainerStarted","Data":"15dc8f7c45f09365012549c864d0d286641c1c6ea28253d2b9feed2fa75a49c1"} Jul 05 19:31:18.909269 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:18.909233 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:22.729792 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:22.729673 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:22.902125 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:22.902017 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8f8zn" event={"ID":"68db1a0c-9074-4478-83cf-e9a5b517b553","Type":"ContainerStarted","Data":"d2f9a2175a133fa970302b7c7bcbc353cdf570224f848c7cc50a13d5edf4bdfd"} Jul 05 19:31:23.348934 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.348890 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bc6b94b7f-8746p"] Jul 05 19:31:23.353536 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.353511 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.356378 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.356351 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 05 19:31:23.357779 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.357760 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 05 19:31:23.357916 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.357804 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 05 19:31:23.357916 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.357813 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 05 19:31:23.357916 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.357763 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 05 19:31:23.358085 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.357910 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ml255\"" Jul 05 19:31:23.361898 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.361877 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bc6b94b7f-8746p"] Jul 05 19:31:23.439714 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.439621 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-serving-cert\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.439714 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.439685 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-oauth-serving-cert\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.439934 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.439769 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-service-ca\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.439934 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.439831 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4cjq\" (UniqueName: \"kubernetes.io/projected/ae69ccd8-18f9-4d61-8fd7-658559db8b98-kube-api-access-m4cjq\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.439934 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.439914 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-oauth-config\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.440082 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.439951 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-config\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.541127 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.541094 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-oauth-config\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.541323 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.541144 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-config\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.541323 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.541192 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-serving-cert\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.541477 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.541435 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-oauth-serving-cert\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.541536 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.541509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-service-ca\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.542507 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.541538 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4cjq\" (UniqueName: \"kubernetes.io/projected/ae69ccd8-18f9-4d61-8fd7-658559db8b98-kube-api-access-m4cjq\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.542507 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.541983 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-config\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.542507 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.542129 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-oauth-serving-cert\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.542507 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.542285 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-service-ca\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.544338 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.544316 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-serving-cert\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.544456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.544407 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-oauth-config\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.549590 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.549567 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4cjq\" (UniqueName: \"kubernetes.io/projected/ae69ccd8-18f9-4d61-8fd7-658559db8b98-kube-api-access-m4cjq\") pod \"console-6bc6b94b7f-8746p\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.665127 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.664989 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:23.818278 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.818246 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bc6b94b7f-8746p"] Jul 05 19:31:23.821601 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:31:23.821570 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae69ccd8_18f9_4d61_8fd7_658559db8b98.slice/crio-d0465183d41a3b9f90b75ae0a7ff70137f43e479ed1f3a4384848dffd7016c6a WatchSource:0}: Error finding container d0465183d41a3b9f90b75ae0a7ff70137f43e479ed1f3a4384848dffd7016c6a: Status 404 returned error can't find the container with id d0465183d41a3b9f90b75ae0a7ff70137f43e479ed1f3a4384848dffd7016c6a Jul 05 19:31:23.866010 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.865977 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6vhql" Jul 05 19:31:23.908798 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.908715 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc6b94b7f-8746p" event={"ID":"ae69ccd8-18f9-4d61-8fd7-658559db8b98","Type":"ContainerStarted","Data":"d0465183d41a3b9f90b75ae0a7ff70137f43e479ed1f3a4384848dffd7016c6a"} Jul 05 19:31:23.911412 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:23.911311 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8f8zn" event={"ID":"68db1a0c-9074-4478-83cf-e9a5b517b553","Type":"ContainerStarted","Data":"0e8b226c7fa6a2288715dd2615e944046abd8b1fb961ad5d0d054fa5efefd9f9"} Jul 05 19:31:24.853667 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:24.853625 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:24.953942 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:24.953914 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:25.054589 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.054551 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:25.417621 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.417584 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:25.517896 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.517854 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:25.645139 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.645102 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:25.665881 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.665844 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp"] Jul 05 19:31:25.672883 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.672798 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" Jul 05 19:31:25.676253 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.676229 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 05 19:31:25.676395 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.676231 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 05 19:31:25.677601 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.677560 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 05 19:31:25.677965 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.677946 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-brbgw\"" Jul 05 19:31:25.680428 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.680363 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp"] Jul 05 19:31:25.699214 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.699187 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw"] Jul 05 19:31:25.703086 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.703060 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.707720 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.707696 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-dtzh5\"" Jul 05 19:31:25.707720 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.707716 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 05 19:31:25.707912 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.707897 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 05 19:31:25.707994 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.707976 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 05 19:31:25.710640 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.709841 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b48b7"] Jul 05 19:31:25.714956 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.713169 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw"] Jul 05 19:31:25.714956 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.713294 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.715808 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.715788 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 05 19:31:25.716188 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.716169 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dkh7n\"" Jul 05 19:31:25.716950 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.716389 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 05 19:31:25.716950 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.716712 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 05 19:31:25.732546 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.732518 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:25.760603 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.760568 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-xcjkp\" (UID: \"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" Jul 05 19:31:25.760781 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.760629 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-xcjkp\" (UID: \"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" Jul 05 19:31:25.760781 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.760695 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gpc6\" (UniqueName: \"kubernetes.io/projected/ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c-kube-api-access-5gpc6\") pod \"openshift-state-metrics-54cf96dfcf-xcjkp\" (UID: \"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" Jul 05 19:31:25.760781 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.760723 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-xcjkp\" (UID: \"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" Jul 05 19:31:25.862087 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862022 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr5kd\" (UniqueName: \"kubernetes.io/projected/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-kube-api-access-lr5kd\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.862554 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862186 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e44061fd-79c4-4cb4-b98f-0f4d848233c6-node-exporter-wtmp\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.862554 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862306 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e44061fd-79c4-4cb4-b98f-0f4d848233c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.862554 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862393 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfscn\" (UniqueName: \"kubernetes.io/projected/e44061fd-79c4-4cb4-b98f-0f4d848233c6-kube-api-access-qfscn\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.862554 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862473 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.862554 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862500 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.862817 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862556 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gpc6\" (UniqueName: \"kubernetes.io/projected/ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c-kube-api-access-5gpc6\") pod \"openshift-state-metrics-54cf96dfcf-xcjkp\" (UID: \"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" Jul 05 19:31:25.862817 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862581 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e44061fd-79c4-4cb4-b98f-0f4d848233c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.862817 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862636 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-xcjkp\" (UID: \"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" Jul 05 19:31:25.862817 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862661 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e44061fd-79c4-4cb4-b98f-0f4d848233c6-node-exporter-textfile\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.862817 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862710 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e44061fd-79c4-4cb4-b98f-0f4d848233c6-node-exporter-tls\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.862817 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862765 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.862817 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862800 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-xcjkp\" (UID: \"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" Jul 05 19:31:25.863175 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862871 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e44061fd-79c4-4cb4-b98f-0f4d848233c6-sys\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.863175 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862922 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.863175 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.862966 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e44061fd-79c4-4cb4-b98f-0f4d848233c6-root\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.863175 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.863007 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e44061fd-79c4-4cb4-b98f-0f4d848233c6-metrics-client-ca\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.863175 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.863063 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-xcjkp\" (UID: \"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" Jul 05 19:31:25.863175 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.863098 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.863621 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:31:25.863597 2573 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 05 19:31:25.863723 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:31:25.863680 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c-openshift-state-metrics-tls podName:ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c nodeName:}" failed. No retries permitted until 2026-07-05 19:31:26.363658015 +0000 UTC m=+87.363676099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-xcjkp" (UID: "ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c") : secret "openshift-state-metrics-tls" not found Jul 05 19:31:25.864729 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.864699 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-xcjkp\" (UID: \"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" Jul 05 19:31:25.870394 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.870366 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-xcjkp\" (UID: \"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" Jul 05 19:31:25.882340 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.882309 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:25.886530 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.886500 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gpc6\" (UniqueName: \"kubernetes.io/projected/ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c-kube-api-access-5gpc6\") pod \"openshift-state-metrics-54cf96dfcf-xcjkp\" (UID: \"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" Jul 05 19:31:25.964735 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.964646 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.964735 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.964713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e44061fd-79c4-4cb4-b98f-0f4d848233c6-sys\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.964960 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.964751 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.964960 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.964792 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e44061fd-79c4-4cb4-b98f-0f4d848233c6-root\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.964960 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.964816 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e44061fd-79c4-4cb4-b98f-0f4d848233c6-metrics-client-ca\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.964960 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.964866 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.964960 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.964905 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr5kd\" (UniqueName: \"kubernetes.io/projected/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-kube-api-access-lr5kd\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.964960 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.964937 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e44061fd-79c4-4cb4-b98f-0f4d848233c6-node-exporter-wtmp\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.965324 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.964977 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e44061fd-79c4-4cb4-b98f-0f4d848233c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.965324 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.965013 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfscn\" (UniqueName: \"kubernetes.io/projected/e44061fd-79c4-4cb4-b98f-0f4d848233c6-kube-api-access-qfscn\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.965324 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.965087 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.965324 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.965118 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.965324 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.965153 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e44061fd-79c4-4cb4-b98f-0f4d848233c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.965543 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.965515 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e44061fd-79c4-4cb4-b98f-0f4d848233c6-sys\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.967061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.965805 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e44061fd-79c4-4cb4-b98f-0f4d848233c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.967061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.965954 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e44061fd-79c4-4cb4-b98f-0f4d848233c6-node-exporter-wtmp\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.967061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.966225 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.967061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.966289 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e44061fd-79c4-4cb4-b98f-0f4d848233c6-root\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.967061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.966321 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e44061fd-79c4-4cb4-b98f-0f4d848233c6-node-exporter-textfile\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.967061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.966576 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.967061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.966575 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e44061fd-79c4-4cb4-b98f-0f4d848233c6-node-exporter-textfile\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.967061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.966815 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e44061fd-79c4-4cb4-b98f-0f4d848233c6-node-exporter-tls\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.967061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.966975 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e44061fd-79c4-4cb4-b98f-0f4d848233c6-metrics-client-ca\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.968177 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.968153 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.971743 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.971695 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.972791 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.972745 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.973291 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.973265 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e44061fd-79c4-4cb4-b98f-0f4d848233c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.975748 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.975704 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfscn\" (UniqueName: \"kubernetes.io/projected/e44061fd-79c4-4cb4-b98f-0f4d848233c6-kube-api-access-qfscn\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.976447 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.976424 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr5kd\" (UniqueName: \"kubernetes.io/projected/4219ed94-cc22-41e7-8d74-f9f4fbb199ee-kube-api-access-lr5kd\") pod \"kube-state-metrics-7c9dbb8b7d-8mhrw\" (UID: \"4219ed94-cc22-41e7-8d74-f9f4fbb199ee\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:25.977423 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.977395 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e44061fd-79c4-4cb4-b98f-0f4d848233c6-node-exporter-tls\") pod \"node-exporter-b48b7\" (UID: \"e44061fd-79c4-4cb4-b98f-0f4d848233c6\") " pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:25.983339 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:25.983253 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:26.021821 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:26.021789 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" Jul 05 19:31:26.030124 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:26.029915 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b48b7" Jul 05 19:31:26.043198 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:26.043167 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:26.371461 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:26.371408 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-xcjkp\" (UID: \"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" Jul 05 19:31:26.376099 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:26.374732 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-xcjkp\" (UID: \"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" Jul 05 19:31:26.545232 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:26.545187 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:26.587561 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:26.587523 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" Jul 05 19:31:26.651399 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:26.651286 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:26.723100 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:26.723067 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:27.318634 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.318579 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:27.547061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.547020 2573 patch_prober.go:28] interesting pod/image-registry-7575585c8f-8gvjn container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 05 19:31:27.547303 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.547099 2573 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" podUID="8cab1cf3-0844-475a-9a85-45b7621f7be0" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 05 19:31:27.607478 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:31:27.606877 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44061fd_79c4_4cb4_b98f_0f4d848233c6.slice/crio-d8184100748eba798557f5967b364d1b25253d35d251b143427b28781eb63ba0 WatchSource:0}: Error finding container d8184100748eba798557f5967b364d1b25253d35d251b143427b28781eb63ba0: Status 404 returned error can't find the container with id d8184100748eba798557f5967b364d1b25253d35d251b143427b28781eb63ba0 Jul 05 19:31:27.652558 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.652533 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr"] Jul 05 19:31:27.656848 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.656143 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.662950 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.660440 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 05 19:31:27.662950 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.660761 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-c0lh5pc8a2bje\"" Jul 05 19:31:27.662950 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.660946 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 05 19:31:27.662950 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.661145 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 05 19:31:27.662950 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.661352 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-btkck\"" Jul 05 19:31:27.662950 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.661527 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 05 19:31:27.662950 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.661789 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 05 19:31:27.670619 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.670297 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr"] Jul 05 19:31:27.775979 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.775955 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp"] Jul 05 19:31:27.777686 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:31:27.777650 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3b51ff_cd85_43a1_b05b_72f32e2e3c4c.slice/crio-63c1cb8564585753ff2249cbce71d09d296a342166d88700c920ab61546b69cd WatchSource:0}: Error finding container 63c1cb8564585753ff2249cbce71d09d296a342166d88700c920ab61546b69cd: Status 404 returned error can't find the container with id 63c1cb8564585753ff2249cbce71d09d296a342166d88700c920ab61546b69cd Jul 05 19:31:27.789913 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.789880 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b910b1ae-d8e3-4201-a800-06319b8e02bf-metrics-client-ca\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.790026 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.789935 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-grpc-tls\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.790026 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.789971 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.790026 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.790011 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87m86\" (UniqueName: \"kubernetes.io/projected/b910b1ae-d8e3-4201-a800-06319b8e02bf-kube-api-access-87m86\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.790212 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.790069 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.790212 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.790093 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.790212 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.790131 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.790358 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.790217 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-thanos-querier-tls\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.797533 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.797506 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw"] Jul 05 19:31:27.800301 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:31:27.800115 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4219ed94_cc22_41e7_8d74_f9f4fbb199ee.slice/crio-3dff5e4f3d659a131679f596266f28c1b7abc334fe781fc8e817136cf48abee0 WatchSource:0}: Error finding container 3dff5e4f3d659a131679f596266f28c1b7abc334fe781fc8e817136cf48abee0: Status 404 returned error can't find the container with id 3dff5e4f3d659a131679f596266f28c1b7abc334fe781fc8e817136cf48abee0 Jul 05 19:31:27.890731 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.890702 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.890873 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.890746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-thanos-querier-tls\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.890873 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.890774 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b910b1ae-d8e3-4201-a800-06319b8e02bf-metrics-client-ca\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.890873 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.890806 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-grpc-tls\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.890873 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.890832 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.890873 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.890868 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87m86\" (UniqueName: \"kubernetes.io/projected/b910b1ae-d8e3-4201-a800-06319b8e02bf-kube-api-access-87m86\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.891141 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.890900 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.891141 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.890927 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.892110 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.891870 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b910b1ae-d8e3-4201-a800-06319b8e02bf-metrics-client-ca\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.894067 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.893623 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-thanos-querier-tls\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.894067 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.893878 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-grpc-tls\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.894192 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.894154 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.894521 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.894490 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.894635 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.894496 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.894687 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.894625 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b910b1ae-d8e3-4201-a800-06319b8e02bf-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.898454 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.898422 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87m86\" (UniqueName: \"kubernetes.io/projected/b910b1ae-d8e3-4201-a800-06319b8e02bf-kube-api-access-87m86\") pod \"thanos-querier-6f9b4d8c9b-vq7vr\" (UID: \"b910b1ae-d8e3-4201-a800-06319b8e02bf\") " pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:27.932887 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.932846 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b48b7" event={"ID":"e44061fd-79c4-4cb4-b98f-0f4d848233c6","Type":"ContainerStarted","Data":"d8184100748eba798557f5967b364d1b25253d35d251b143427b28781eb63ba0"} Jul 05 19:31:27.934373 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.934344 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" event={"ID":"4219ed94-cc22-41e7-8d74-f9f4fbb199ee","Type":"ContainerStarted","Data":"3dff5e4f3d659a131679f596266f28c1b7abc334fe781fc8e817136cf48abee0"} Jul 05 19:31:27.936072 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.935938 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc6b94b7f-8746p" event={"ID":"ae69ccd8-18f9-4d61-8fd7-658559db8b98","Type":"ContainerStarted","Data":"b800194a4dc8fc3a44a7c4edc888fc205991497bb0b3485ad491901fb855208a"} Jul 05 19:31:27.947748 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.947719 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" event={"ID":"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c","Type":"ContainerStarted","Data":"4f21939f477e43e92c3688a605800394df25895dece9f83a0500082ed0972e23"} Jul 05 19:31:27.947893 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.947790 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" event={"ID":"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c","Type":"ContainerStarted","Data":"25f4782cdcdabf6cbf9aebc134788b1daffadbaf566a426a05be057877db7753"} Jul 05 19:31:27.947893 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.947810 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" event={"ID":"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c","Type":"ContainerStarted","Data":"63c1cb8564585753ff2249cbce71d09d296a342166d88700c920ab61546b69cd"} Jul 05 19:31:27.950412 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.950385 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8f8zn" event={"ID":"68db1a0c-9074-4478-83cf-e9a5b517b553","Type":"ContainerStarted","Data":"09849904ebe6f4366d8bd491552aa02aebb0161c84c819dad45fecd705f05abe"} Jul 05 19:31:27.956493 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.956453 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bc6b94b7f-8746p" podStartSLOduration=1.168210442 podStartE2EDuration="4.956438914s" totalImagesPullingTime="3.788228472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:27.954659057 +0000 UTC m=+88.954677164" watchObservedRunningTime="2026-07-05 19:31:27.956438914 +0000 UTC m=+88.956457020" Jul 05 19:31:27.973113 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.973003 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8f8zn" podStartSLOduration=32.890195901 podStartE2EDuration="41.972986385s" totalImagesPullingTime="9.082790484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 19:30:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:27.971556867 +0000 UTC m=+88.971574983" watchObservedRunningTime="2026-07-05 19:31:27.972986385 +0000 UTC m=+88.973004491" Jul 05 19:31:27.995894 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:27.995860 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:28.148901 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:28.148871 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr"] Jul 05 19:31:28.573142 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:28.573105 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:28.673583 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:28.673543 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:28.774058 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:28.774011 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:28.831903 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:28.831824 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:30.339129 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:30.339096 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:30.419271 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:30.419200 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:30.485958 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:30.485916 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:30.823316 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:30.823288 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:31:31.076550 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.076387 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:31.176412 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.176358 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:31.231937 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.231906 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:31.810810 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.810761 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:31.812686 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.812664 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 19:31:31.816542 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.816523 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.819435 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.819411 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 05 19:31:31.819571 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.819488 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 05 19:31:31.819642 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.819589 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 05 19:31:31.820115 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.819818 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zgn5j\"" Jul 05 19:31:31.820115 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.819851 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 05 19:31:31.820115 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.819861 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 05 19:31:31.820115 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.819863 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 05 19:31:31.820742 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.820722 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9d86jrpgfkac6\"" Jul 05 19:31:31.820742 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.820736 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 05 19:31:31.820957 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.820767 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 05 19:31:31.820957 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.820877 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 05 19:31:31.820957 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.820727 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 05 19:31:31.821555 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.821537 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 05 19:31:31.821839 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.821820 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 05 19:31:31.822008 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.821996 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 05 19:31:31.822213 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.822199 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 05 19:31:31.823812 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.823795 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 05 19:31:31.830324 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.830259 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.830324 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.830298 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.830473 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.830348 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.830473 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.830386 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dd98ea6-676c-4f9c-9878-5ca542516697-config-out\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.830585 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.830503 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.830768 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.830708 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.830871 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.830793 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-web-config\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.830871 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.830824 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.830871 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.830850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.831024 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.830878 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.831024 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.830903 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.831024 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.830927 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.831024 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.830950 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.831024 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.830976 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.831024 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.831003 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.831326 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.831033 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-config\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.831326 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.831079 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.831326 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.831119 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dd98ea6-676c-4f9c-9878-5ca542516697-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.831326 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.831143 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.831326 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.831168 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvsw\" (UniqueName: \"kubernetes.io/projected/9dd98ea6-676c-4f9c-9878-5ca542516697-kube-api-access-9jvsw\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.832468 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.832445 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 19:31:31.925248 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.925211 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:31.932259 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932228 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-web-config\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932259 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932262 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932478 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932288 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932478 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932478 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932448 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932625 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932478 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932625 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932503 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932625 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932537 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932625 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932567 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932625 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932607 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-config\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932851 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932630 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932851 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932675 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dd98ea6-676c-4f9c-9878-5ca542516697-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932851 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932709 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932851 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932734 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvsw\" (UniqueName: \"kubernetes.io/projected/9dd98ea6-676c-4f9c-9878-5ca542516697-kube-api-access-9jvsw\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932851 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932787 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932851 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932811 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.932851 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.933214 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932889 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dd98ea6-676c-4f9c-9878-5ca542516697-config-out\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.933214 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932919 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.933214 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932934 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.933214 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.932949 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.934028 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.933363 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.934028 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.933569 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.934028 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.933951 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.935104 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.934769 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.935266 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.935240 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.935401 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.935354 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.936781 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.936757 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.936945 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.936891 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.937929 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.937428 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.937929 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.937805 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.937929 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.937928 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.938753 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.938733 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.938935 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.938916 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dd98ea6-676c-4f9c-9878-5ca542516697-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.939340 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.939321 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dd98ea6-676c-4f9c-9878-5ca542516697-config-out\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.939474 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.939357 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-web-config\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.939608 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.939584 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.939658 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.939595 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-config\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.940962 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.940939 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:31.947697 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:31.947678 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvsw\" (UniqueName: \"kubernetes.io/projected/9dd98ea6-676c-4f9c-9878-5ca542516697-kube-api-access-9jvsw\") pod \"prometheus-k8s-0\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:32.025894 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:32.025854 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:32.129087 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:32.128991 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:32.982452 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:32.982413 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:33.036209 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:33.036169 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:33.557801 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:33.557765 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7575585c8f-8gvjn"] Jul 05 19:31:33.635329 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:33.635285 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:33.665382 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:33.665349 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:33.665561 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:33.665411 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:33.673190 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:33.673164 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:33.695562 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:33.695519 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:33.763010 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:33.762967 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:33.864480 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:33.864394 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:33.984662 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:33.984621 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:31:34.260455 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:34.260420 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:34.316579 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:34.316549 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:35.128080 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:31:35.128030 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb910b1ae_d8e3_4201_a800_06319b8e02bf.slice/crio-ca9865a3050451d514c67e9d28faa2c6b3cbfce248a7ded82a76c1c6dddc0e87 WatchSource:0}: Error finding container ca9865a3050451d514c67e9d28faa2c6b3cbfce248a7ded82a76c1c6dddc0e87: Status 404 returned error can't find the container with id ca9865a3050451d514c67e9d28faa2c6b3cbfce248a7ded82a76c1c6dddc0e87 Jul 05 19:31:35.267750 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:35.267719 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 19:31:35.269109 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:31:35.269086 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd98ea6_676c_4f9c_9878_5ca542516697.slice/crio-206f9f8b9d7cc7360b1c8fa882993ab0b2b8fc59be846f76f9efb79729f13446 WatchSource:0}: Error finding container 206f9f8b9d7cc7360b1c8fa882993ab0b2b8fc59be846f76f9efb79729f13446: Status 404 returned error can't find the container with id 206f9f8b9d7cc7360b1c8fa882993ab0b2b8fc59be846f76f9efb79729f13446 Jul 05 19:31:35.996575 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:35.996486 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" event={"ID":"b910b1ae-d8e3-4201-a800-06319b8e02bf","Type":"ContainerStarted","Data":"ca9865a3050451d514c67e9d28faa2c6b3cbfce248a7ded82a76c1c6dddc0e87"} Jul 05 19:31:35.999679 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:35.999644 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-p6dfl" event={"ID":"fa57f98a-83b8-4dfb-8c4e-97126b6b9cbe","Type":"ContainerStarted","Data":"078eef065be0c3f6b197968c87f9d654f905ede94f1acb45d6de83d4348a6daf"} Jul 05 19:31:36.001211 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:36.000716 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-p6dfl" Jul 05 19:31:36.005453 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:36.005431 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-p6dfl" Jul 05 19:31:36.010839 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:36.010799 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dd98ea6-676c-4f9c-9878-5ca542516697","Type":"ContainerStarted","Data":"206f9f8b9d7cc7360b1c8fa882993ab0b2b8fc59be846f76f9efb79729f13446"} Jul 05 19:31:36.018968 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:36.018904 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-p6dfl" podStartSLOduration=0.951726491 podStartE2EDuration="22.018885362s" totalImagesPullingTime="21.067158871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:36.016398602 +0000 UTC m=+97.016416708" watchObservedRunningTime="2026-07-05 19:31:36.018885362 +0000 UTC m=+97.018903468" Jul 05 19:31:36.129904 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:36.129853 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:36.229869 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:36.229831 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:37.790959 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:37.790931 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qsk8d" Jul 05 19:31:39.112457 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:39.112407 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:39.214065 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:39.214015 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:39.314438 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:39.314397 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:39.777179 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:31:39.774976 2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd98ea6_676c_4f9c_9878_5ca542516697.slice/crio-conmon-f2bb74f3ed54e7f2d8474ad9109a8ceb3d4af103dd631b752d28bff11f30df72.scope\": RecentStats: unable to find data in memory cache]" Jul 05 19:31:40.044373 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:40.044334 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" event={"ID":"b910b1ae-d8e3-4201-a800-06319b8e02bf","Type":"ContainerStarted","Data":"d3b656e9b3d48314b8bb85335ab525d3481d790ee26b395f1eba46b702322bd4"} Jul 05 19:31:40.044544 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:40.044474 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" event={"ID":"b910b1ae-d8e3-4201-a800-06319b8e02bf","Type":"ContainerStarted","Data":"804804f2daaea6aee2a30b30296986548ea9d99ee75fa2ee82184b463e54ae54"} Jul 05 19:31:40.044544 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:40.044535 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" event={"ID":"b910b1ae-d8e3-4201-a800-06319b8e02bf","Type":"ContainerStarted","Data":"bf721fb573190636065fbc4b687947c90edc49df5348ff79dbb3e5ec3f6688c7"} Jul 05 19:31:40.046771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:40.046739 2573 generic.go:356] "Generic (PLEG): container finished" podID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerID="f2bb74f3ed54e7f2d8474ad9109a8ceb3d4af103dd631b752d28bff11f30df72" exitCode=0 Jul 05 19:31:40.046884 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:40.046845 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dd98ea6-676c-4f9c-9878-5ca542516697","Type":"ContainerDied","Data":"f2bb74f3ed54e7f2d8474ad9109a8ceb3d4af103dd631b752d28bff11f30df72"} Jul 05 19:31:40.050624 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:40.050598 2573 generic.go:356] "Generic (PLEG): container finished" podID="e44061fd-79c4-4cb4-b98f-0f4d848233c6" containerID="5ad40905b743b58f80a0281f9f7ad1485f35201126c3b2482c8de05821c2b4b6" exitCode=0 Jul 05 19:31:40.050724 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:40.050673 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b48b7" event={"ID":"e44061fd-79c4-4cb4-b98f-0f4d848233c6","Type":"ContainerDied","Data":"5ad40905b743b58f80a0281f9f7ad1485f35201126c3b2482c8de05821c2b4b6"} Jul 05 19:31:40.057933 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:40.057906 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" event={"ID":"4219ed94-cc22-41e7-8d74-f9f4fbb199ee","Type":"ContainerStarted","Data":"fc9d33cb193f93714de1977a4c50df46e4c8b14786b3ee5ddc16af465b3b024a"} Jul 05 19:31:40.058061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:40.057966 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" event={"ID":"4219ed94-cc22-41e7-8d74-f9f4fbb199ee","Type":"ContainerStarted","Data":"80844bad2f9f4a058f3fd08052ff518383f19b73f0d3dfcb340f014fb7efeb48"} Jul 05 19:31:40.058061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:40.057997 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" event={"ID":"4219ed94-cc22-41e7-8d74-f9f4fbb199ee","Type":"ContainerStarted","Data":"c14c3a211b1ebf29ec37bc9e1261aa91c3c07c5ac9ce53560e89432b4e344900"} Jul 05 19:31:40.061156 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:40.061111 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" event={"ID":"ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c","Type":"ContainerStarted","Data":"38e383c723f691535b9722e0e212d8bc7fdd08a595dd7e3c2d1884722a8d69eb"} Jul 05 19:31:40.095129 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:40.095069 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-xcjkp" podStartSLOduration=3.54608013 podStartE2EDuration="15.095030083s" totalImagesPullingTime="11.548949953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:40.093469867 +0000 UTC m=+101.093487975" watchObservedRunningTime="2026-07-05 19:31:40.095030083 +0000 UTC m=+101.095048190" Jul 05 19:31:40.134920 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:40.134859 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-8mhrw" podStartSLOduration=3.474800698 podStartE2EDuration="15.134836882s" totalImagesPullingTime="11.660036184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:40.133624037 +0000 UTC m=+101.133642143" watchObservedRunningTime="2026-07-05 19:31:40.134836882 +0000 UTC m=+101.134854987" Jul 05 19:31:41.071382 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:41.071341 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b48b7" event={"ID":"e44061fd-79c4-4cb4-b98f-0f4d848233c6","Type":"ContainerStarted","Data":"cfb4b62c3205584f58dc60182e9f27c512aa0ce6513df2335912850a44037345"} Jul 05 19:31:41.071797 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:41.071414 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b48b7" event={"ID":"e44061fd-79c4-4cb4-b98f-0f4d848233c6","Type":"ContainerStarted","Data":"f1c682af090c84264f30a031fb40391c10350c47336d80ce44132262953e704c"} Jul 05 19:31:41.092122 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:41.092059 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b48b7" podStartSLOduration=4.084221329 podStartE2EDuration="16.092023599s" totalImagesPullingTime="11.850816394s" totalInitContainerRuntime="156.985876ms" isStatefulPod=true podCreationTimestamp="2026-07-05 19:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:41.090813769 +0000 UTC m=+102.090831879" watchObservedRunningTime="2026-07-05 19:31:41.092023599 +0000 UTC m=+102.092041703" Jul 05 19:31:41.998152 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:41.998122 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:42.083872 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:42.082990 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" event={"ID":"b910b1ae-d8e3-4201-a800-06319b8e02bf","Type":"ContainerStarted","Data":"c3205ea90da24f621b0b7e4bab049a0370726286e788d0ac10b9d8935e1b4d40"} Jul 05 19:31:42.083872 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:42.083097 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" event={"ID":"b910b1ae-d8e3-4201-a800-06319b8e02bf","Type":"ContainerStarted","Data":"2a3392164c30bb075c81be1f4a210717f0687ede221075c5831cf6bda2c142e2"} Jul 05 19:31:42.083872 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:42.083144 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" event={"ID":"b910b1ae-d8e3-4201-a800-06319b8e02bf","Type":"ContainerStarted","Data":"a8bb47833197327533b6449dc429224a08c04d74945018de9aec87015aebde30"} Jul 05 19:31:42.098922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:42.098828 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:42.108496 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:42.108432 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" podStartSLOduration=9.369031919 podStartE2EDuration="15.108409942s" totalImagesPullingTime="5.739378023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:31:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:42.107901675 +0000 UTC m=+103.107919805" watchObservedRunningTime="2026-07-05 19:31:42.108409942 +0000 UTC m=+103.108428048" Jul 05 19:31:42.708881 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:42.708838 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bc6b94b7f-8746p"] Jul 05 19:31:42.769016 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:42.768910 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:42.867831 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:42.867791 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:43.093334 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:43.093298 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:43.588031 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:43.587991 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:43.688516 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:43.688063 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:44.104585 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:44.104551 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6f9b4d8c9b-vq7vr" Jul 05 19:31:44.552307 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:44.552270 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:44.634662 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:44.634583 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:45.104032 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:45.103996 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dd98ea6-676c-4f9c-9878-5ca542516697","Type":"ContainerStarted","Data":"0308c385a5561783cb0a773bd599f26f55aaa54fea230c7a869ce0007ff1ef6f"} Jul 05 19:31:45.104156 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:45.104118 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dd98ea6-676c-4f9c-9878-5ca542516697","Type":"ContainerStarted","Data":"d12c0057219887ec48fc1867dbf8446d70b569829f676777d9f4e14bc8c3f70c"} Jul 05 19:31:45.104251 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:45.104162 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dd98ea6-676c-4f9c-9878-5ca542516697","Type":"ContainerStarted","Data":"b0f9ae0b1d046004be240e6fe38147b5bcf654256da414dc47b83a8cf339849a"} Jul 05 19:31:45.104251 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:45.104199 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dd98ea6-676c-4f9c-9878-5ca542516697","Type":"ContainerStarted","Data":"b07942ca07f48f40a84c865ce9333bd75d7d9d152719a4be41a497cc82758c96"} Jul 05 19:31:45.269337 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:45.269308 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:45.334448 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:45.334363 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:46.115580 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:46.115539 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dd98ea6-676c-4f9c-9878-5ca542516697","Type":"ContainerStarted","Data":"ed42bea6d0d2b1d263c5bc23238bffede87942b771950fb8e102a21dc186da51"} Jul 05 19:31:46.115766 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:46.115651 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dd98ea6-676c-4f9c-9878-5ca542516697","Type":"ContainerStarted","Data":"50fd2b65eb4004da62e770b123e088585de209806469e04e5d1d5700eaf6d7ae"} Jul 05 19:31:46.144410 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:46.144345 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.457348109 podStartE2EDuration="15.144324873s" totalImagesPullingTime="8.64433351s" totalInitContainerRuntime="42.643254ms" isStatefulPod=false podCreationTimestamp="2026-07-05 19:31:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:31:46.142118289 +0000 UTC m=+107.142136395" watchObservedRunningTime="2026-07-05 19:31:46.144324873 +0000 UTC m=+107.144342979" Jul 05 19:31:47.129867 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:47.129830 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:31:57.160331 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:57.160293 2573 generic.go:356] "Generic (PLEG): container finished" podID="81eba20f-cd1e-41cb-a868-ee8f133a20ef" containerID="ca8fca1ea75bca4f15d2be986c9408048cc5617f8dd89171b4ed97edfde3fba6" exitCode=0 Jul 05 19:31:57.160809 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:57.160361 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" event={"ID":"81eba20f-cd1e-41cb-a868-ee8f133a20ef","Type":"ContainerDied","Data":"ca8fca1ea75bca4f15d2be986c9408048cc5617f8dd89171b4ed97edfde3fba6"} Jul 05 19:31:57.160809 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:57.160624 2573 scope.go:117] "RemoveContainer" containerID="ca8fca1ea75bca4f15d2be986c9408048cc5617f8dd89171b4ed97edfde3fba6" Jul 05 19:31:57.597516 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:57.597479 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:57.656330 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:57.656282 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:31:58.165891 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.165860 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-k8pzp" event={"ID":"81eba20f-cd1e-41cb-a868-ee8f133a20ef","Type":"ContainerStarted","Data":"a2b523c45c2b4ba69465651fb08a92a42db5ce9da84768d674474b23b62846ad"} Jul 05 19:31:58.587311 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.587278 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" podUID="8cab1cf3-0844-475a-9a85-45b7621f7be0" containerName="registry" containerID="cri-o://e5be2000613131b780fa4be6b25f1dd4e6d9016b264a7eb50d91bed17ff55730" gracePeriod=30 Jul 05 19:31:58.860109 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.860026 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:31:58.995620 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.995585 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cab1cf3-0844-475a-9a85-45b7621f7be0-trusted-ca\") pod \"8cab1cf3-0844-475a-9a85-45b7621f7be0\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " Jul 05 19:31:58.995793 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.995645 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls\") pod \"8cab1cf3-0844-475a-9a85-45b7621f7be0\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " Jul 05 19:31:58.995793 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.995682 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cab1cf3-0844-475a-9a85-45b7621f7be0-image-registry-private-configuration\") pod \"8cab1cf3-0844-475a-9a85-45b7621f7be0\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " Jul 05 19:31:58.995793 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.995697 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-bound-sa-token\") pod \"8cab1cf3-0844-475a-9a85-45b7621f7be0\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " Jul 05 19:31:58.995793 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.995729 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cab1cf3-0844-475a-9a85-45b7621f7be0-ca-trust-extracted\") pod \"8cab1cf3-0844-475a-9a85-45b7621f7be0\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " Jul 05 19:31:58.995986 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.995815 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppgqz\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-kube-api-access-ppgqz\") pod \"8cab1cf3-0844-475a-9a85-45b7621f7be0\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " Jul 05 19:31:58.995986 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.995840 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-certificates\") pod \"8cab1cf3-0844-475a-9a85-45b7621f7be0\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " Jul 05 19:31:58.995986 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.995867 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cab1cf3-0844-475a-9a85-45b7621f7be0-installation-pull-secrets\") pod \"8cab1cf3-0844-475a-9a85-45b7621f7be0\" (UID: \"8cab1cf3-0844-475a-9a85-45b7621f7be0\") " Jul 05 19:31:58.996227 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.996085 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cab1cf3-0844-475a-9a85-45b7621f7be0-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8cab1cf3-0844-475a-9a85-45b7621f7be0" (UID: "8cab1cf3-0844-475a-9a85-45b7621f7be0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:31:58.996675 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.996598 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8cab1cf3-0844-475a-9a85-45b7621f7be0" (UID: "8cab1cf3-0844-475a-9a85-45b7621f7be0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:31:58.998524 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.998486 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cab1cf3-0844-475a-9a85-45b7621f7be0-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8cab1cf3-0844-475a-9a85-45b7621f7be0" (UID: "8cab1cf3-0844-475a-9a85-45b7621f7be0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:31:58.998640 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.998539 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-kube-api-access-ppgqz" (OuterVolumeSpecName: "kube-api-access-ppgqz") pod "8cab1cf3-0844-475a-9a85-45b7621f7be0" (UID: "8cab1cf3-0844-475a-9a85-45b7621f7be0"). InnerVolumeSpecName "kube-api-access-ppgqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:31:58.998713 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.998662 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cab1cf3-0844-475a-9a85-45b7621f7be0-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8cab1cf3-0844-475a-9a85-45b7621f7be0" (UID: "8cab1cf3-0844-475a-9a85-45b7621f7be0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:31:58.998713 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.998662 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8cab1cf3-0844-475a-9a85-45b7621f7be0" (UID: "8cab1cf3-0844-475a-9a85-45b7621f7be0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:31:58.999000 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:58.998963 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8cab1cf3-0844-475a-9a85-45b7621f7be0" (UID: "8cab1cf3-0844-475a-9a85-45b7621f7be0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:31:59.004637 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.004596 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cab1cf3-0844-475a-9a85-45b7621f7be0-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8cab1cf3-0844-475a-9a85-45b7621f7be0" (UID: "8cab1cf3-0844-475a-9a85-45b7621f7be0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:31:59.097009 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.096969 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppgqz\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-kube-api-access-ppgqz\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:31:59.097009 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.097008 2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-certificates\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:31:59.097230 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.097022 2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cab1cf3-0844-475a-9a85-45b7621f7be0-installation-pull-secrets\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:31:59.097230 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.097036 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cab1cf3-0844-475a-9a85-45b7621f7be0-trusted-ca\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:31:59.097230 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.097075 2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-registry-tls\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:31:59.097230 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.097089 2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cab1cf3-0844-475a-9a85-45b7621f7be0-image-registry-private-configuration\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:31:59.097230 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.097102 2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cab1cf3-0844-475a-9a85-45b7621f7be0-bound-sa-token\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:31:59.097230 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.097116 2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cab1cf3-0844-475a-9a85-45b7621f7be0-ca-trust-extracted\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:31:59.170927 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.170829 2573 generic.go:356] "Generic (PLEG): container finished" podID="8cab1cf3-0844-475a-9a85-45b7621f7be0" containerID="e5be2000613131b780fa4be6b25f1dd4e6d9016b264a7eb50d91bed17ff55730" exitCode=0 Jul 05 19:31:59.170927 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.170868 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" event={"ID":"8cab1cf3-0844-475a-9a85-45b7621f7be0","Type":"ContainerDied","Data":"e5be2000613131b780fa4be6b25f1dd4e6d9016b264a7eb50d91bed17ff55730"} Jul 05 19:31:59.170927 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.170886 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" Jul 05 19:31:59.170927 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.170902 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7575585c8f-8gvjn" event={"ID":"8cab1cf3-0844-475a-9a85-45b7621f7be0","Type":"ContainerDied","Data":"0c778c7db50d9bd99ea22fab12ae92cf276cd6150f3e57a24a9ecbe81265aadf"} Jul 05 19:31:59.171543 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.170925 2573 scope.go:117] "RemoveContainer" containerID="e5be2000613131b780fa4be6b25f1dd4e6d9016b264a7eb50d91bed17ff55730" Jul 05 19:31:59.179930 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.179911 2573 scope.go:117] "RemoveContainer" containerID="e5be2000613131b780fa4be6b25f1dd4e6d9016b264a7eb50d91bed17ff55730" Jul 05 19:31:59.180195 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:31:59.180175 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5be2000613131b780fa4be6b25f1dd4e6d9016b264a7eb50d91bed17ff55730\": container with ID starting with e5be2000613131b780fa4be6b25f1dd4e6d9016b264a7eb50d91bed17ff55730 not found: ID does not exist" containerID="e5be2000613131b780fa4be6b25f1dd4e6d9016b264a7eb50d91bed17ff55730" Jul 05 19:31:59.180240 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.180201 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5be2000613131b780fa4be6b25f1dd4e6d9016b264a7eb50d91bed17ff55730"} err="failed to get container status \"e5be2000613131b780fa4be6b25f1dd4e6d9016b264a7eb50d91bed17ff55730\": rpc error: code = NotFound desc = could not find container \"e5be2000613131b780fa4be6b25f1dd4e6d9016b264a7eb50d91bed17ff55730\": container with ID starting with e5be2000613131b780fa4be6b25f1dd4e6d9016b264a7eb50d91bed17ff55730 not found: ID does not exist" Jul 05 19:31:59.192399 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.192374 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7575585c8f-8gvjn"] Jul 05 19:31:59.193729 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.193710 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7575585c8f-8gvjn"] Jul 05 19:31:59.469668 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:31:59.469596 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cab1cf3-0844-475a-9a85-45b7621f7be0" path="/var/lib/kubelet/pods/8cab1cf3-0844-475a-9a85-45b7621f7be0/volumes" Jul 05 19:32:01.179225 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:01.179188 2573 generic.go:356] "Generic (PLEG): container finished" podID="bc38af11-062b-458d-8f23-4d34f9fd1fdd" containerID="5f128f4d977082d1b4145ef91a005e1380862f2b29b9b50786a9cfc25bd46840" exitCode=0 Jul 05 19:32:01.179594 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:01.179259 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" event={"ID":"bc38af11-062b-458d-8f23-4d34f9fd1fdd","Type":"ContainerDied","Data":"5f128f4d977082d1b4145ef91a005e1380862f2b29b9b50786a9cfc25bd46840"} Jul 05 19:32:01.179643 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:01.179632 2573 scope.go:117] "RemoveContainer" containerID="5f128f4d977082d1b4145ef91a005e1380862f2b29b9b50786a9cfc25bd46840" Jul 05 19:32:02.185702 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:02.185672 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-wz8gp" event={"ID":"bc38af11-062b-458d-8f23-4d34f9fd1fdd","Type":"ContainerStarted","Data":"56305ee63574f2c2d647601efdc60daa9307acef7ff2e502a6d0d2e6a40f5779"} Jul 05 19:32:07.746410 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:07.746336 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6bc6b94b7f-8746p" podUID="ae69ccd8-18f9-4d61-8fd7-658559db8b98" containerName="console" containerID="cri-o://b800194a4dc8fc3a44a7c4edc888fc205991497bb0b3485ad491901fb855208a" gracePeriod=15 Jul 05 19:32:08.029223 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.029192 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:32:08.176415 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.176376 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-config\") pod \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " Jul 05 19:32:08.176586 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.176425 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-oauth-config\") pod \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " Jul 05 19:32:08.176586 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.176446 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-service-ca\") pod \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " Jul 05 19:32:08.176586 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.176470 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-serving-cert\") pod \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " Jul 05 19:32:08.176586 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.176521 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4cjq\" (UniqueName: \"kubernetes.io/projected/ae69ccd8-18f9-4d61-8fd7-658559db8b98-kube-api-access-m4cjq\") pod \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " Jul 05 19:32:08.176586 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.176541 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-oauth-serving-cert\") pod \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\" (UID: \"ae69ccd8-18f9-4d61-8fd7-658559db8b98\") " Jul 05 19:32:08.176944 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.176901 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-service-ca" (OuterVolumeSpecName: "service-ca") pod "ae69ccd8-18f9-4d61-8fd7-658559db8b98" (UID: "ae69ccd8-18f9-4d61-8fd7-658559db8b98"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:32:08.176944 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.176924 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-config" (OuterVolumeSpecName: "console-config") pod "ae69ccd8-18f9-4d61-8fd7-658559db8b98" (UID: "ae69ccd8-18f9-4d61-8fd7-658559db8b98"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:32:08.177148 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.176981 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ae69ccd8-18f9-4d61-8fd7-658559db8b98" (UID: "ae69ccd8-18f9-4d61-8fd7-658559db8b98"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:32:08.178915 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.178886 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ae69ccd8-18f9-4d61-8fd7-658559db8b98" (UID: "ae69ccd8-18f9-4d61-8fd7-658559db8b98"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:08.179001 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.178928 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae69ccd8-18f9-4d61-8fd7-658559db8b98-kube-api-access-m4cjq" (OuterVolumeSpecName: "kube-api-access-m4cjq") pod "ae69ccd8-18f9-4d61-8fd7-658559db8b98" (UID: "ae69ccd8-18f9-4d61-8fd7-658559db8b98"). InnerVolumeSpecName "kube-api-access-m4cjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:32:08.179001 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.178970 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ae69ccd8-18f9-4d61-8fd7-658559db8b98" (UID: "ae69ccd8-18f9-4d61-8fd7-658559db8b98"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:08.210958 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.210925 2573 generic.go:356] "Generic (PLEG): container finished" podID="ae69ccd8-18f9-4d61-8fd7-658559db8b98" containerID="b800194a4dc8fc3a44a7c4edc888fc205991497bb0b3485ad491901fb855208a" exitCode=0 Jul 05 19:32:08.211116 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.210974 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc6b94b7f-8746p" Jul 05 19:32:08.211116 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.211013 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc6b94b7f-8746p" event={"ID":"ae69ccd8-18f9-4d61-8fd7-658559db8b98","Type":"ContainerDied","Data":"b800194a4dc8fc3a44a7c4edc888fc205991497bb0b3485ad491901fb855208a"} Jul 05 19:32:08.211116 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.211096 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc6b94b7f-8746p" event={"ID":"ae69ccd8-18f9-4d61-8fd7-658559db8b98","Type":"ContainerDied","Data":"d0465183d41a3b9f90b75ae0a7ff70137f43e479ed1f3a4384848dffd7016c6a"} Jul 05 19:32:08.211226 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.211125 2573 scope.go:117] "RemoveContainer" containerID="b800194a4dc8fc3a44a7c4edc888fc205991497bb0b3485ad491901fb855208a" Jul 05 19:32:08.219749 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.219733 2573 scope.go:117] "RemoveContainer" containerID="b800194a4dc8fc3a44a7c4edc888fc205991497bb0b3485ad491901fb855208a" Jul 05 19:32:08.220007 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:32:08.219985 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b800194a4dc8fc3a44a7c4edc888fc205991497bb0b3485ad491901fb855208a\": container with ID starting with b800194a4dc8fc3a44a7c4edc888fc205991497bb0b3485ad491901fb855208a not found: ID does not exist" containerID="b800194a4dc8fc3a44a7c4edc888fc205991497bb0b3485ad491901fb855208a" Jul 05 19:32:08.220119 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.220012 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b800194a4dc8fc3a44a7c4edc888fc205991497bb0b3485ad491901fb855208a"} err="failed to get container status \"b800194a4dc8fc3a44a7c4edc888fc205991497bb0b3485ad491901fb855208a\": rpc error: code = NotFound desc = could not find container \"b800194a4dc8fc3a44a7c4edc888fc205991497bb0b3485ad491901fb855208a\": container with ID starting with b800194a4dc8fc3a44a7c4edc888fc205991497bb0b3485ad491901fb855208a not found: ID does not exist" Jul 05 19:32:08.236022 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.235993 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bc6b94b7f-8746p"] Jul 05 19:32:08.240446 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.240418 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bc6b94b7f-8746p"] Jul 05 19:32:08.278119 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.278017 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-config\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:08.278119 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.278057 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-oauth-config\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:08.278119 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.278068 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-service-ca\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:08.278119 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.278079 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ae69ccd8-18f9-4d61-8fd7-658559db8b98-console-serving-cert\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:08.278119 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.278088 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4cjq\" (UniqueName: \"kubernetes.io/projected/ae69ccd8-18f9-4d61-8fd7-658559db8b98-kube-api-access-m4cjq\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:08.278119 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:08.278096 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ae69ccd8-18f9-4d61-8fd7-658559db8b98-oauth-serving-cert\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:09.469655 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:09.469619 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae69ccd8-18f9-4d61-8fd7-658559db8b98" path="/var/lib/kubelet/pods/ae69ccd8-18f9-4d61-8fd7-658559db8b98/volumes" Jul 05 19:32:12.596345 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:12.596311 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:32:12.649229 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:12.649190 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:32:17.254397 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:17.254360 2573 generic.go:356] "Generic (PLEG): container finished" podID="1071794b-4f80-4a07-a571-e790c329a568" containerID="2bf557a08904aac3bd51f5ba12f2761d53e593d80a7c896e8c0e3ad26ec40d91" exitCode=0 Jul 05 19:32:17.254985 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:17.254422 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" event={"ID":"1071794b-4f80-4a07-a571-e790c329a568","Type":"ContainerDied","Data":"2bf557a08904aac3bd51f5ba12f2761d53e593d80a7c896e8c0e3ad26ec40d91"} Jul 05 19:32:17.254985 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:17.254780 2573 scope.go:117] "RemoveContainer" containerID="2bf557a08904aac3bd51f5ba12f2761d53e593d80a7c896e8c0e3ad26ec40d91" Jul 05 19:32:18.260239 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:18.260202 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-tn8sz" event={"ID":"1071794b-4f80-4a07-a571-e790c329a568","Type":"ContainerStarted","Data":"ee01327cab00f959b432bd971a055ac703854896b2880c86255c309acbff9953"} Jul 05 19:32:32.129844 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:32.129796 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:32.146707 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:32.146674 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:32.334138 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:32.334109 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:44.807208 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:44.807171 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:32:45.235439 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:45.235336 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:32:50.185251 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.185210 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 19:32:50.185698 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.185553 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="kube-rbac-proxy" containerID="cri-o://50fd2b65eb4004da62e770b123e088585de209806469e04e5d1d5700eaf6d7ae" gracePeriod=600 Jul 05 19:32:50.185698 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.185582 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="thanos-sidecar" containerID="cri-o://d12c0057219887ec48fc1867dbf8446d70b569829f676777d9f4e14bc8c3f70c" gracePeriod=600 Jul 05 19:32:50.185698 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.185599 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="kube-rbac-proxy-web" containerID="cri-o://0308c385a5561783cb0a773bd599f26f55aaa54fea230c7a869ce0007ff1ef6f" gracePeriod=600 Jul 05 19:32:50.185698 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.185538 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="prometheus" containerID="cri-o://b07942ca07f48f40a84c865ce9333bd75d7d9d152719a4be41a497cc82758c96" gracePeriod=600 Jul 05 19:32:50.185698 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.185626 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="config-reloader" containerID="cri-o://b0f9ae0b1d046004be240e6fe38147b5bcf654256da414dc47b83a8cf339849a" gracePeriod=600 Jul 05 19:32:50.185698 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.185654 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ed42bea6d0d2b1d263c5bc23238bffede87942b771950fb8e102a21dc186da51" gracePeriod=600 Jul 05 19:32:50.396771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.396736 2573 generic.go:356] "Generic (PLEG): container finished" podID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerID="ed42bea6d0d2b1d263c5bc23238bffede87942b771950fb8e102a21dc186da51" exitCode=0 Jul 05 19:32:50.396771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.396765 2573 generic.go:356] "Generic (PLEG): container finished" podID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerID="50fd2b65eb4004da62e770b123e088585de209806469e04e5d1d5700eaf6d7ae" exitCode=0 Jul 05 19:32:50.396771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.396772 2573 generic.go:356] "Generic (PLEG): container finished" podID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerID="0308c385a5561783cb0a773bd599f26f55aaa54fea230c7a869ce0007ff1ef6f" exitCode=0 Jul 05 19:32:50.396771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.396778 2573 generic.go:356] "Generic (PLEG): container finished" podID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerID="d12c0057219887ec48fc1867dbf8446d70b569829f676777d9f4e14bc8c3f70c" exitCode=0 Jul 05 19:32:50.396771 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.396783 2573 generic.go:356] "Generic (PLEG): container finished" podID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerID="b0f9ae0b1d046004be240e6fe38147b5bcf654256da414dc47b83a8cf339849a" exitCode=0 Jul 05 19:32:50.397104 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.396790 2573 generic.go:356] "Generic (PLEG): container finished" podID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerID="b07942ca07f48f40a84c865ce9333bd75d7d9d152719a4be41a497cc82758c96" exitCode=0 Jul 05 19:32:50.397104 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.396802 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dd98ea6-676c-4f9c-9878-5ca542516697","Type":"ContainerDied","Data":"ed42bea6d0d2b1d263c5bc23238bffede87942b771950fb8e102a21dc186da51"} Jul 05 19:32:50.397104 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.396893 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dd98ea6-676c-4f9c-9878-5ca542516697","Type":"ContainerDied","Data":"50fd2b65eb4004da62e770b123e088585de209806469e04e5d1d5700eaf6d7ae"} Jul 05 19:32:50.397104 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.396920 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dd98ea6-676c-4f9c-9878-5ca542516697","Type":"ContainerDied","Data":"0308c385a5561783cb0a773bd599f26f55aaa54fea230c7a869ce0007ff1ef6f"} Jul 05 19:32:50.397104 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.396943 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dd98ea6-676c-4f9c-9878-5ca542516697","Type":"ContainerDied","Data":"d12c0057219887ec48fc1867dbf8446d70b569829f676777d9f4e14bc8c3f70c"} Jul 05 19:32:50.397104 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.396969 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dd98ea6-676c-4f9c-9878-5ca542516697","Type":"ContainerDied","Data":"b0f9ae0b1d046004be240e6fe38147b5bcf654256da414dc47b83a8cf339849a"} Jul 05 19:32:50.397104 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.397004 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dd98ea6-676c-4f9c-9878-5ca542516697","Type":"ContainerDied","Data":"b07942ca07f48f40a84c865ce9333bd75d7d9d152719a4be41a497cc82758c96"} Jul 05 19:32:50.433296 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.433273 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:50.549088 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549017 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-thanos-prometheus-http-client-file\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549088 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549094 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-0\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549359 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549141 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-web-config\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549359 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549185 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-db\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549359 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549219 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-2\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549359 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549246 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549359 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549278 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-1\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549359 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549300 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549359 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549324 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-tls\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549359 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549350 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-serving-certs-ca-bundle\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549751 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549376 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-metrics-client-ca\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549751 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549401 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-kube-rbac-proxy\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549751 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549431 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-grpc-tls\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549751 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549460 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dd98ea6-676c-4f9c-9878-5ca542516697-tls-assets\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549751 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549498 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-metrics-client-certs\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549751 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549534 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-config\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549751 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549574 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-trusted-ca-bundle\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549751 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549605 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-kubelet-serving-ca-bundle\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549751 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549611 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:32:50.549751 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549641 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dd98ea6-676c-4f9c-9878-5ca542516697-config-out\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.549751 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549665 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jvsw\" (UniqueName: \"kubernetes.io/projected/9dd98ea6-676c-4f9c-9878-5ca542516697-kube-api-access-9jvsw\") pod \"9dd98ea6-676c-4f9c-9878-5ca542516697\" (UID: \"9dd98ea6-676c-4f9c-9878-5ca542516697\") " Jul 05 19:32:50.551913 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549870 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:32:50.551913 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.549930 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.551913 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.550383 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:32:50.551913 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.551061 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:32:50.551913 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.551330 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:32:50.551913 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.551339 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:32:50.551913 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.551501 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:32:50.551913 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.551512 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:32:50.552759 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.552724 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:50.553091 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.553016 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:50.553599 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.553568 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:50.553889 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.553864 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-config" (OuterVolumeSpecName: "config") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:50.554242 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.554205 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:50.554456 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.554430 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:50.554558 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.554450 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:50.554654 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.554635 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd98ea6-676c-4f9c-9878-5ca542516697-kube-api-access-9jvsw" (OuterVolumeSpecName: "kube-api-access-9jvsw") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "kube-api-access-9jvsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:32:50.554946 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.554930 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:50.555194 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.555179 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd98ea6-676c-4f9c-9878-5ca542516697-config-out" (OuterVolumeSpecName: "config-out") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:32:50.555336 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.555323 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd98ea6-676c-4f9c-9878-5ca542516697-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:32:50.564985 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.564961 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-web-config" (OuterVolumeSpecName: "web-config") pod "9dd98ea6-676c-4f9c-9878-5ca542516697" (UID: "9dd98ea6-676c-4f9c-9878-5ca542516697"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:32:50.650341 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650311 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-db\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650341 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650340 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650351 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650360 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650369 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650378 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650387 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-metrics-client-ca\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650395 2573 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-kube-rbac-proxy\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650403 2573 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-grpc-tls\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650411 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dd98ea6-676c-4f9c-9878-5ca542516697-tls-assets\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650418 2573 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-secret-metrics-client-certs\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650426 2573 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-config\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650434 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650441 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650449 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dd98ea6-676c-4f9c-9878-5ca542516697-config-out\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650456 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jvsw\" (UniqueName: \"kubernetes.io/projected/9dd98ea6-676c-4f9c-9878-5ca542516697-kube-api-access-9jvsw\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650464 2573 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650474 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9dd98ea6-676c-4f9c-9878-5ca542516697-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:50.650506 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:50.650501 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dd98ea6-676c-4f9c-9878-5ca542516697-web-config\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:32:51.404612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.404575 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9dd98ea6-676c-4f9c-9878-5ca542516697","Type":"ContainerDied","Data":"206f9f8b9d7cc7360b1c8fa882993ab0b2b8fc59be846f76f9efb79729f13446"} Jul 05 19:32:51.404612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.404600 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.405117 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.404668 2573 scope.go:117] "RemoveContainer" containerID="ed42bea6d0d2b1d263c5bc23238bffede87942b771950fb8e102a21dc186da51" Jul 05 19:32:51.414034 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.413946 2573 scope.go:117] "RemoveContainer" containerID="50fd2b65eb4004da62e770b123e088585de209806469e04e5d1d5700eaf6d7ae" Jul 05 19:32:51.422062 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.422029 2573 scope.go:117] "RemoveContainer" containerID="0308c385a5561783cb0a773bd599f26f55aaa54fea230c7a869ce0007ff1ef6f" Jul 05 19:32:51.429803 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.429785 2573 scope.go:117] "RemoveContainer" containerID="d12c0057219887ec48fc1867dbf8446d70b569829f676777d9f4e14bc8c3f70c" Jul 05 19:32:51.431743 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.431716 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 19:32:51.437317 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.437295 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 19:32:51.437727 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.437714 2573 scope.go:117] "RemoveContainer" containerID="b0f9ae0b1d046004be240e6fe38147b5bcf654256da414dc47b83a8cf339849a" Jul 05 19:32:51.444973 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.444955 2573 scope.go:117] "RemoveContainer" containerID="b07942ca07f48f40a84c865ce9333bd75d7d9d152719a4be41a497cc82758c96" Jul 05 19:32:51.452996 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.452970 2573 scope.go:117] "RemoveContainer" containerID="f2bb74f3ed54e7f2d8474ad9109a8ceb3d4af103dd631b752d28bff11f30df72" Jul 05 19:32:51.463246 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463221 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 19:32:51.463592 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463574 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="thanos-sidecar" Jul 05 19:32:51.463669 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463595 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="thanos-sidecar" Jul 05 19:32:51.463669 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463626 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="kube-rbac-proxy" Jul 05 19:32:51.463669 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463636 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="kube-rbac-proxy" Jul 05 19:32:51.463669 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463653 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cab1cf3-0844-475a-9a85-45b7621f7be0" containerName="registry" Jul 05 19:32:51.463669 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463661 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cab1cf3-0844-475a-9a85-45b7621f7be0" containerName="registry" Jul 05 19:32:51.463669 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463669 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="prometheus" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463677 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="prometheus" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463686 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="init-config-reloader" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463695 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="init-config-reloader" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463704 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="kube-rbac-proxy-web" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463713 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="kube-rbac-proxy-web" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463730 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="kube-rbac-proxy-thanos" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463739 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="kube-rbac-proxy-thanos" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463750 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae69ccd8-18f9-4d61-8fd7-658559db8b98" containerName="console" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463758 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae69ccd8-18f9-4d61-8fd7-658559db8b98" containerName="console" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463766 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="config-reloader" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463774 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="config-reloader" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463847 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="kube-rbac-proxy" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463860 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="kube-rbac-proxy-web" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463870 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae69ccd8-18f9-4d61-8fd7-658559db8b98" containerName="console" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463881 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="thanos-sidecar" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463890 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="prometheus" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463897 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="kube-rbac-proxy-thanos" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463906 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cab1cf3-0844-475a-9a85-45b7621f7be0" containerName="registry" Jul 05 19:32:51.463966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.463913 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" containerName="config-reloader" Jul 05 19:32:51.468286 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.468266 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.471404 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.471383 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 05 19:32:51.471493 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.471407 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 05 19:32:51.471586 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.471568 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 05 19:32:51.471703 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.471685 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 05 19:32:51.471876 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.471858 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 05 19:32:51.471949 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.471876 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 05 19:32:51.471949 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.471896 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zgn5j\"" Jul 05 19:32:51.471949 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.471920 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 05 19:32:51.472121 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.472108 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 05 19:32:51.472384 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.472366 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 05 19:32:51.472997 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.472972 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 05 19:32:51.473107 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.473031 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9d86jrpgfkac6\"" Jul 05 19:32:51.473107 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.473093 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 05 19:32:51.473256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.473235 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 05 19:32:51.473368 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.473352 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 05 19:32:51.473936 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.473913 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dd98ea6-676c-4f9c-9878-5ca542516697" path="/var/lib/kubelet/pods/9dd98ea6-676c-4f9c-9878-5ca542516697/volumes" Jul 05 19:32:51.475204 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.475188 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 05 19:32:51.478116 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.478096 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 05 19:32:51.483293 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.483272 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 19:32:51.558238 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558198 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558428 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558241 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558428 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558272 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558428 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558428 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558373 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-config\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558428 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558391 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558428 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558412 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558658 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558483 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-web-config\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558658 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558543 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558658 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558577 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2e2692b-152b-4686-b5fc-13549d58457d-config-out\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558658 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558596 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b2e2692b-152b-4686-b5fc-13549d58457d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558658 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558611 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558658 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558627 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558658 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558647 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558892 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558681 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558892 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558698 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558892 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558713 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558892 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558733 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2e2692b-152b-4686-b5fc-13549d58457d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558892 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558753 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh5k9\" (UniqueName: \"kubernetes.io/projected/b2e2692b-152b-4686-b5fc-13549d58457d-kube-api-access-bh5k9\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.558892 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.558804 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659222 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659222 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659225 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-config\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659478 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659478 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659373 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659593 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659503 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-web-config\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659593 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659550 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659690 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659598 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2e2692b-152b-4686-b5fc-13549d58457d-config-out\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659690 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659630 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b2e2692b-152b-4686-b5fc-13549d58457d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659690 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659656 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659690 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659681 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659883 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659715 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659883 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659751 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659883 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659782 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659883 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659806 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659883 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659840 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2e2692b-152b-4686-b5fc-13549d58457d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.659883 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659865 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh5k9\" (UniqueName: \"kubernetes.io/projected/b2e2692b-152b-4686-b5fc-13549d58457d-kube-api-access-bh5k9\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.660376 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659888 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.660376 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659910 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.660376 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659964 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.660376 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.659989 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.660376 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.660018 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.660376 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.660019 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.662287 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.662203 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-config\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.662656 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.662596 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.662656 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.662639 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.662793 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.662696 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-web-config\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.664441 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.662887 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b2e2692b-152b-4686-b5fc-13549d58457d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.664441 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.663281 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.664441 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.663702 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.664441 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.663894 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.664441 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.664090 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.664441 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.664382 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.665290 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.664870 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2e2692b-152b-4686-b5fc-13549d58457d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.665290 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.665002 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.665440 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.665421 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2e2692b-152b-4686-b5fc-13549d58457d-config-out\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.665499 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.665467 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2e2692b-152b-4686-b5fc-13549d58457d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.665561 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.665534 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.665561 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.665546 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.666207 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.666153 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b2e2692b-152b-4686-b5fc-13549d58457d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.668495 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.668474 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh5k9\" (UniqueName: \"kubernetes.io/projected/b2e2692b-152b-4686-b5fc-13549d58457d-kube-api-access-bh5k9\") pod \"prometheus-k8s-0\" (UID: \"b2e2692b-152b-4686-b5fc-13549d58457d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.779278 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.779226 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:32:51.927938 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:51.927911 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 19:32:51.930168 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:32:51.930140 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e2692b_152b_4686_b5fc_13549d58457d.slice/crio-521ea1ab36f217426198217d8d20f5364014dde7a17c29ae81a78e7aa3b2aeee WatchSource:0}: Error finding container 521ea1ab36f217426198217d8d20f5364014dde7a17c29ae81a78e7aa3b2aeee: Status 404 returned error can't find the container with id 521ea1ab36f217426198217d8d20f5364014dde7a17c29ae81a78e7aa3b2aeee Jul 05 19:32:52.410480 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:52.410449 2573 generic.go:356] "Generic (PLEG): container finished" podID="b2e2692b-152b-4686-b5fc-13549d58457d" containerID="8792a72c191ac9855ee1071e378d044ea719bdd511269f6d186b698267453897" exitCode=0 Jul 05 19:32:52.410959 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:52.410545 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2e2692b-152b-4686-b5fc-13549d58457d","Type":"ContainerDied","Data":"8792a72c191ac9855ee1071e378d044ea719bdd511269f6d186b698267453897"} Jul 05 19:32:52.410959 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:52.410639 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2e2692b-152b-4686-b5fc-13549d58457d","Type":"ContainerStarted","Data":"521ea1ab36f217426198217d8d20f5364014dde7a17c29ae81a78e7aa3b2aeee"} Jul 05 19:32:53.419437 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:53.419401 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2e2692b-152b-4686-b5fc-13549d58457d","Type":"ContainerStarted","Data":"78e7732493a11da0811f8454de2aa2fc11cfd36d1a4bc3f15f03ba55cc9b73c8"} Jul 05 19:32:53.419810 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:53.419498 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2e2692b-152b-4686-b5fc-13549d58457d","Type":"ContainerStarted","Data":"c116cb56f09392e2122bbf4b1d0df732d0edda8f4d070a097542a798b2ab2429"} Jul 05 19:32:53.419810 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:53.419526 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2e2692b-152b-4686-b5fc-13549d58457d","Type":"ContainerStarted","Data":"3d7723133ce7b17bab69c4cf4478af94b94ff1d36025efaf4fb9319697e41eb1"} Jul 05 19:32:53.419810 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:53.419559 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2e2692b-152b-4686-b5fc-13549d58457d","Type":"ContainerStarted","Data":"198f6f0a27c036393a9d728a9ecdbd09789a9fb482bdd43cda4dfc9019d09a81"} Jul 05 19:32:53.419810 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:53.419584 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2e2692b-152b-4686-b5fc-13549d58457d","Type":"ContainerStarted","Data":"ca7b4f45b25b28804789f1251a974bdb300fe9fca947e15a4e84442d59938d5a"} Jul 05 19:32:53.419810 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:53.419608 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b2e2692b-152b-4686-b5fc-13549d58457d","Type":"ContainerStarted","Data":"f91731c9a716c0c67351aeb9264e46c664b391abf93940f723a050bdf5f92698"} Jul 05 19:32:53.449029 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:53.448974 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.429177756 podStartE2EDuration="2.448954759s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.777003ms" isStatefulPod=false podCreationTimestamp="2026-07-05 19:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:32:53.446437708 +0000 UTC m=+174.446455848" watchObservedRunningTime="2026-07-05 19:32:53.448954759 +0000 UTC m=+174.448972864" Jul 05 19:32:53.716671 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:53.716576 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:32:53.769653 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:53.769621 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:32:53.863686 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:53.863614 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:32:53.963264 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:53.963224 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:32:54.460926 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:54.460890 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:32:54.515913 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:54.515872 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:32:56.780065 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:32:56.779989 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:33:05.152655 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:33:05.152623 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:33:05.257002 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:33:05.256951 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:33:05.354179 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:33:05.354140 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:33:31.889736 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:33:31.889700 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:33:31.942262 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:33:31.942228 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:33:51.780379 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:33:51.780336 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:33:51.796643 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:33:51.796613 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:33:52.679188 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:33:52.679158 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 19:34:00.012944 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:00.012906 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:34:00.071276 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:00.071243 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:34:06.180673 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:06.180612 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:34:06.313933 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:06.313902 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:34:06.376366 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:06.376330 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:34:06.919713 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:06.919677 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:34:07.018615 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:07.018577 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:34:11.931469 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:11.931420 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:34:12.032125 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:12.032087 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:34:17.382331 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:17.382291 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:34:17.490032 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:17.489995 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:34:27.615159 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:27.615007 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:34:27.717324 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:27.717275 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:34:59.379088 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:59.379031 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:34:59.417789 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:59.417759 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 19:34:59.422638 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:59.422611 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 19:34:59.427904 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:34:59.427888 2573 kubelet.go:1648] "Image garbage collection succeeded" Jul 05 19:35:00.643833 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:00.643799 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:16.710445 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:16.710408 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:26.101015 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:26.100984 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:26.933995 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:26.933957 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:29.723672 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:29.723640 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:29.825121 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:29.825085 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:31.412445 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:31.412403 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:31.506926 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:31.506874 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:31.965256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:31.965216 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:32.070503 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:32.070461 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:33.407785 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:33.407742 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:33.799396 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:33.799366 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:36.713986 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:36.713950 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:36.816457 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:36.816413 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:42.004898 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:42.004865 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:43.544209 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:43.544172 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:43.621928 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:43.621890 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:54.448596 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:54.448489 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:54.545821 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:54.545775 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:59.006028 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:59.005993 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:35:59.105451 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:35:59.105419 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:01.058055 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:01.058020 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:14.814253 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:14.814210 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:14.902693 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:14.902655 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:15.328399 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.328361 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:15.401314 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.401278 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv"] Jul 05 19:36:15.403621 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.403596 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:15.406972 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.406951 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 05 19:36:15.408466 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.408446 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-k6bnp\"" Jul 05 19:36:15.408466 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.408451 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 05 19:36:15.408643 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.408475 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 05 19:36:15.408643 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.408482 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 05 19:36:15.408643 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.408451 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 05 19:36:15.418675 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.418655 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv"] Jul 05 19:36:15.430510 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.430484 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:15.488798 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.488760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t48s4\" (UniqueName: \"kubernetes.io/projected/05a48ac2-0f73-4290-b6c2-f3e0a7262452-kube-api-access-t48s4\") pod \"lws-controller-manager-db9cc5d86-fgqmv\" (UID: \"05a48ac2-0f73-4290-b6c2-f3e0a7262452\") " pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:15.488798 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.488803 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05a48ac2-0f73-4290-b6c2-f3e0a7262452-cert\") pod \"lws-controller-manager-db9cc5d86-fgqmv\" (UID: \"05a48ac2-0f73-4290-b6c2-f3e0a7262452\") " pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:15.488997 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.488954 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/05a48ac2-0f73-4290-b6c2-f3e0a7262452-metrics-cert\") pod \"lws-controller-manager-db9cc5d86-fgqmv\" (UID: \"05a48ac2-0f73-4290-b6c2-f3e0a7262452\") " pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:15.489066 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.488999 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/05a48ac2-0f73-4290-b6c2-f3e0a7262452-manager-config\") pod \"lws-controller-manager-db9cc5d86-fgqmv\" (UID: \"05a48ac2-0f73-4290-b6c2-f3e0a7262452\") " pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:15.553169 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.553141 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:15.589966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.589887 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/05a48ac2-0f73-4290-b6c2-f3e0a7262452-metrics-cert\") pod \"lws-controller-manager-db9cc5d86-fgqmv\" (UID: \"05a48ac2-0f73-4290-b6c2-f3e0a7262452\") " pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:15.589966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.589925 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/05a48ac2-0f73-4290-b6c2-f3e0a7262452-manager-config\") pod \"lws-controller-manager-db9cc5d86-fgqmv\" (UID: \"05a48ac2-0f73-4290-b6c2-f3e0a7262452\") " pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:15.589966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.589950 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t48s4\" (UniqueName: \"kubernetes.io/projected/05a48ac2-0f73-4290-b6c2-f3e0a7262452-kube-api-access-t48s4\") pod \"lws-controller-manager-db9cc5d86-fgqmv\" (UID: \"05a48ac2-0f73-4290-b6c2-f3e0a7262452\") " pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:15.590260 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.590087 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05a48ac2-0f73-4290-b6c2-f3e0a7262452-cert\") pod \"lws-controller-manager-db9cc5d86-fgqmv\" (UID: \"05a48ac2-0f73-4290-b6c2-f3e0a7262452\") " pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:15.590661 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.590637 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/05a48ac2-0f73-4290-b6c2-f3e0a7262452-manager-config\") pod \"lws-controller-manager-db9cc5d86-fgqmv\" (UID: \"05a48ac2-0f73-4290-b6c2-f3e0a7262452\") " pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:15.592624 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.592603 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05a48ac2-0f73-4290-b6c2-f3e0a7262452-cert\") pod \"lws-controller-manager-db9cc5d86-fgqmv\" (UID: \"05a48ac2-0f73-4290-b6c2-f3e0a7262452\") " pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:15.592700 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.592666 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/05a48ac2-0f73-4290-b6c2-f3e0a7262452-metrics-cert\") pod \"lws-controller-manager-db9cc5d86-fgqmv\" (UID: \"05a48ac2-0f73-4290-b6c2-f3e0a7262452\") " pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:15.601185 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.601156 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t48s4\" (UniqueName: \"kubernetes.io/projected/05a48ac2-0f73-4290-b6c2-f3e0a7262452-kube-api-access-t48s4\") pod \"lws-controller-manager-db9cc5d86-fgqmv\" (UID: \"05a48ac2-0f73-4290-b6c2-f3e0a7262452\") " pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:15.712999 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.712949 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:15.816662 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.816617 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:15.879337 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.879312 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv"] Jul 05 19:36:15.881872 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:36:15.881848 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a48ac2_0f73_4290_b6c2_f3e0a7262452.slice/crio-439a9008654f105156d59331f322a8ee37809e1fa86ec61e6b43c4fe8b454ea8 WatchSource:0}: Error finding container 439a9008654f105156d59331f322a8ee37809e1fa86ec61e6b43c4fe8b454ea8: Status 404 returned error can't find the container with id 439a9008654f105156d59331f322a8ee37809e1fa86ec61e6b43c4fe8b454ea8 Jul 05 19:36:15.883736 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.883719 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 05 19:36:15.936115 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:15.936078 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:16.049085 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.049031 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq"] Jul 05 19:36:16.058083 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.058054 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.061877 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.061851 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k85hm\"" Jul 05 19:36:16.062117 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.062102 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 05 19:36:16.065900 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.065878 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 05 19:36:16.067819 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.067802 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 05 19:36:16.071393 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.071379 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 05 19:36:16.087843 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.087817 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq"] Jul 05 19:36:16.095420 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.095393 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62953dea-e74e-4fe8-a427-6a8a462f4287-tmp\") pod \"opendatahub-operator-controller-manager-68674bcb-mx6pq\" (UID: \"62953dea-e74e-4fe8-a427-6a8a462f4287\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.095557 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.095429 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62953dea-e74e-4fe8-a427-6a8a462f4287-webhook-cert\") pod \"opendatahub-operator-controller-manager-68674bcb-mx6pq\" (UID: \"62953dea-e74e-4fe8-a427-6a8a462f4287\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.095557 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.095487 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62953dea-e74e-4fe8-a427-6a8a462f4287-manifests\") pod \"opendatahub-operator-controller-manager-68674bcb-mx6pq\" (UID: \"62953dea-e74e-4fe8-a427-6a8a462f4287\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.095557 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.095505 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/62953dea-e74e-4fe8-a427-6a8a462f4287-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68674bcb-mx6pq\" (UID: \"62953dea-e74e-4fe8-a427-6a8a462f4287\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.095557 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.095521 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54j9t\" (UniqueName: \"kubernetes.io/projected/62953dea-e74e-4fe8-a427-6a8a462f4287-kube-api-access-54j9t\") pod \"opendatahub-operator-controller-manager-68674bcb-mx6pq\" (UID: \"62953dea-e74e-4fe8-a427-6a8a462f4287\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.196209 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.196115 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62953dea-e74e-4fe8-a427-6a8a462f4287-tmp\") pod \"opendatahub-operator-controller-manager-68674bcb-mx6pq\" (UID: \"62953dea-e74e-4fe8-a427-6a8a462f4287\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.196209 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.196166 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62953dea-e74e-4fe8-a427-6a8a462f4287-webhook-cert\") pod \"opendatahub-operator-controller-manager-68674bcb-mx6pq\" (UID: \"62953dea-e74e-4fe8-a427-6a8a462f4287\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.196433 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.196258 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62953dea-e74e-4fe8-a427-6a8a462f4287-manifests\") pod \"opendatahub-operator-controller-manager-68674bcb-mx6pq\" (UID: \"62953dea-e74e-4fe8-a427-6a8a462f4287\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.196433 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.196282 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/62953dea-e74e-4fe8-a427-6a8a462f4287-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68674bcb-mx6pq\" (UID: \"62953dea-e74e-4fe8-a427-6a8a462f4287\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.196433 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.196305 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54j9t\" (UniqueName: \"kubernetes.io/projected/62953dea-e74e-4fe8-a427-6a8a462f4287-kube-api-access-54j9t\") pod \"opendatahub-operator-controller-manager-68674bcb-mx6pq\" (UID: \"62953dea-e74e-4fe8-a427-6a8a462f4287\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.197463 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.197100 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62953dea-e74e-4fe8-a427-6a8a462f4287-manifests\") pod \"opendatahub-operator-controller-manager-68674bcb-mx6pq\" (UID: \"62953dea-e74e-4fe8-a427-6a8a462f4287\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.201105 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.201075 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/62953dea-e74e-4fe8-a427-6a8a462f4287-webhook-cert\") pod \"opendatahub-operator-controller-manager-68674bcb-mx6pq\" (UID: \"62953dea-e74e-4fe8-a427-6a8a462f4287\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.201728 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.201627 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/62953dea-e74e-4fe8-a427-6a8a462f4287-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68674bcb-mx6pq\" (UID: \"62953dea-e74e-4fe8-a427-6a8a462f4287\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.201850 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.201735 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:16.201922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.201908 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62953dea-e74e-4fe8-a427-6a8a462f4287-tmp\") pod \"opendatahub-operator-controller-manager-68674bcb-mx6pq\" (UID: \"62953dea-e74e-4fe8-a427-6a8a462f4287\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.205210 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.205192 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54j9t\" (UniqueName: \"kubernetes.io/projected/62953dea-e74e-4fe8-a427-6a8a462f4287-kube-api-access-54j9t\") pod \"opendatahub-operator-controller-manager-68674bcb-mx6pq\" (UID: \"62953dea-e74e-4fe8-a427-6a8a462f4287\") " pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.248367 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.248332 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" event={"ID":"05a48ac2-0f73-4290-b6c2-f3e0a7262452","Type":"ContainerStarted","Data":"439a9008654f105156d59331f322a8ee37809e1fa86ec61e6b43c4fe8b454ea8"} Jul 05 19:36:16.300348 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.300318 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:16.369175 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.369143 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:16.400815 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.400773 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:16.499879 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.499848 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:16.511065 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.511026 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq"] Jul 05 19:36:16.691294 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.691261 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:16.784898 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.784864 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:16.886182 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.886129 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:16.989941 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:16.989888 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:17.260134 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:17.260092 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" event={"ID":"62953dea-e74e-4fe8-a427-6a8a462f4287","Type":"ContainerStarted","Data":"91f0a56866d4e6cea3c8ddb88638f49809332b5cd14d7f714e63d09241803317"} Jul 05 19:36:21.281570 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:21.281537 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" event={"ID":"05a48ac2-0f73-4290-b6c2-f3e0a7262452","Type":"ContainerStarted","Data":"a6c62b80340e18b49578159922252d201bb8b404e6f4533f1d4425509ab4e7f4"} Jul 05 19:36:21.282013 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:21.281637 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:21.283243 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:21.283219 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" event={"ID":"62953dea-e74e-4fe8-a427-6a8a462f4287","Type":"ContainerStarted","Data":"c88723dcc5762fc16a9da0bf89f05392b69d273d0f6a50ad35b1bf58f2d54963"} Jul 05 19:36:21.505169 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:21.505118 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" podStartSLOduration=2.122937577 podStartE2EDuration="6.505101683s" totalImagesPullingTime="4.382164106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:36:21.403287929 +0000 UTC m=+382.403306037" watchObservedRunningTime="2026-07-05 19:36:21.505101683 +0000 UTC m=+382.505119790" Jul 05 19:36:22.291441 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:22.291410 2573 generic.go:356] "Generic (PLEG): container finished" podID="62953dea-e74e-4fe8-a427-6a8a462f4287" containerID="c88723dcc5762fc16a9da0bf89f05392b69d273d0f6a50ad35b1bf58f2d54963" exitCode=0 Jul 05 19:36:22.291897 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:22.291506 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" event={"ID":"62953dea-e74e-4fe8-a427-6a8a462f4287","Type":"ContainerDied","Data":"c88723dcc5762fc16a9da0bf89f05392b69d273d0f6a50ad35b1bf58f2d54963"} Jul 05 19:36:23.297748 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:23.297714 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" event={"ID":"62953dea-e74e-4fe8-a427-6a8a462f4287","Type":"ContainerStarted","Data":"cab7a53547861d4273adcf35a2d7e867cd591c230fd9e1ad2fd7add8c8056783"} Jul 05 19:36:23.298169 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:23.297815 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:23.322849 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:23.322792 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" podStartSLOduration=3.069370115 podStartE2EDuration="8.322776161s" totalImagesPullingTime="4.109417236s" totalInitContainerRuntime="1.14398881s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:36:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:36:23.320531481 +0000 UTC m=+384.320549630" watchObservedRunningTime="2026-07-05 19:36:23.322776161 +0000 UTC m=+384.322794266" Jul 05 19:36:27.863381 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:27.863339 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:27.961997 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:27.961968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:32.294401 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:32.294366 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-db9cc5d86-fgqmv" Jul 05 19:36:34.303873 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:34.303844 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68674bcb-mx6pq" Jul 05 19:36:35.841334 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:35.841295 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:35.948432 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:35.948398 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:37.212479 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:37.212445 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:41.507273 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.507216 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:41.600874 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.600842 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7484578f88-bj4ds"] Jul 05 19:36:41.616141 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.616110 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7484578f88-bj4ds" Jul 05 19:36:41.619963 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.619936 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 05 19:36:41.620130 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.619986 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w598p\"" Jul 05 19:36:41.620130 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.620098 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 05 19:36:41.620276 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.620245 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7484578f88-bj4ds"] Jul 05 19:36:41.683860 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.683823 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:41.723156 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.723118 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d3e615-ce5d-44a1-a62e-a43485f7c0ba-tls-certs\") pod \"kube-auth-proxy-7484578f88-bj4ds\" (UID: \"f6d3e615-ce5d-44a1-a62e-a43485f7c0ba\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-bj4ds" Jul 05 19:36:41.723320 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.723171 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk5gt\" (UniqueName: \"kubernetes.io/projected/f6d3e615-ce5d-44a1-a62e-a43485f7c0ba-kube-api-access-nk5gt\") pod \"kube-auth-proxy-7484578f88-bj4ds\" (UID: \"f6d3e615-ce5d-44a1-a62e-a43485f7c0ba\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-bj4ds" Jul 05 19:36:41.723320 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.723191 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6d3e615-ce5d-44a1-a62e-a43485f7c0ba-tmp\") pod \"kube-auth-proxy-7484578f88-bj4ds\" (UID: \"f6d3e615-ce5d-44a1-a62e-a43485f7c0ba\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-bj4ds" Jul 05 19:36:41.818610 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.818505 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:41.823736 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.823711 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk5gt\" (UniqueName: \"kubernetes.io/projected/f6d3e615-ce5d-44a1-a62e-a43485f7c0ba-kube-api-access-nk5gt\") pod \"kube-auth-proxy-7484578f88-bj4ds\" (UID: \"f6d3e615-ce5d-44a1-a62e-a43485f7c0ba\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-bj4ds" Jul 05 19:36:41.823892 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.823744 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6d3e615-ce5d-44a1-a62e-a43485f7c0ba-tmp\") pod \"kube-auth-proxy-7484578f88-bj4ds\" (UID: \"f6d3e615-ce5d-44a1-a62e-a43485f7c0ba\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-bj4ds" Jul 05 19:36:41.823892 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.823804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d3e615-ce5d-44a1-a62e-a43485f7c0ba-tls-certs\") pod \"kube-auth-proxy-7484578f88-bj4ds\" (UID: \"f6d3e615-ce5d-44a1-a62e-a43485f7c0ba\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-bj4ds" Jul 05 19:36:41.826187 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.826167 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6d3e615-ce5d-44a1-a62e-a43485f7c0ba-tmp\") pod \"kube-auth-proxy-7484578f88-bj4ds\" (UID: \"f6d3e615-ce5d-44a1-a62e-a43485f7c0ba\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-bj4ds" Jul 05 19:36:41.826302 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.826286 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6d3e615-ce5d-44a1-a62e-a43485f7c0ba-tls-certs\") pod \"kube-auth-proxy-7484578f88-bj4ds\" (UID: \"f6d3e615-ce5d-44a1-a62e-a43485f7c0ba\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-bj4ds" Jul 05 19:36:41.837150 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.837117 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk5gt\" (UniqueName: \"kubernetes.io/projected/f6d3e615-ce5d-44a1-a62e-a43485f7c0ba-kube-api-access-nk5gt\") pod \"kube-auth-proxy-7484578f88-bj4ds\" (UID: \"f6d3e615-ce5d-44a1-a62e-a43485f7c0ba\") " pod="openshift-ingress/kube-auth-proxy-7484578f88-bj4ds" Jul 05 19:36:41.927759 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.927726 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7484578f88-bj4ds" Jul 05 19:36:41.932033 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:41.931993 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:42.032905 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:42.032870 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:42.067110 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:42.067079 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7484578f88-bj4ds"] Jul 05 19:36:42.069857 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:36:42.069800 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d3e615_ce5d_44a1_a62e_a43485f7c0ba.slice/crio-8f06f8b1415e7e62b4fdfc6b0e300250c36000e08c042618d8e56b1675de5e07 WatchSource:0}: Error finding container 8f06f8b1415e7e62b4fdfc6b0e300250c36000e08c042618d8e56b1675de5e07: Status 404 returned error can't find the container with id 8f06f8b1415e7e62b4fdfc6b0e300250c36000e08c042618d8e56b1675de5e07 Jul 05 19:36:42.173734 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:42.173703 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:42.381887 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:42.381802 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7484578f88-bj4ds" event={"ID":"f6d3e615-ce5d-44a1-a62e-a43485f7c0ba","Type":"ContainerStarted","Data":"8f06f8b1415e7e62b4fdfc6b0e300250c36000e08c042618d8e56b1675de5e07"} Jul 05 19:36:42.499256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:42.499220 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:42.653220 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:42.653120 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:43.042913 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:43.042875 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:43.108916 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:43.108843 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:43.890701 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:43.890665 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:43.957203 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:43.957163 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:44.172646 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:44.172561 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:45.397313 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:45.397283 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7484578f88-bj4ds" event={"ID":"f6d3e615-ce5d-44a1-a62e-a43485f7c0ba","Type":"ContainerStarted","Data":"c680ba12ca6bfb303aef9356811c9029d6ec8b375920f37de341d7f50cf84494"} Jul 05 19:36:45.413951 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:45.413898 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7484578f88-bj4ds" podStartSLOduration=1.808766981 podStartE2EDuration="4.41387707s" totalImagesPullingTime="2.605110089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:36:45.413576541 +0000 UTC m=+406.413594661" watchObservedRunningTime="2026-07-05 19:36:45.41387707 +0000 UTC m=+406.413895177" Jul 05 19:36:49.213553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:49.213514 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:49.328082 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:49.328031 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:49.968810 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:49.968776 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:50.170939 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:50.170900 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:50.276221 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:50.276180 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:50.400202 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:50.400166 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:53.079901 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:53.079866 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:53.283599 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:53.283554 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:53.417704 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:53.417616 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:54.448985 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:54.448949 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:36:54.548363 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:36:54.548330 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:37:37.303677 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:37:37.303585 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:37:37.406385 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:37:37.406348 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:37:37.479944 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:37:37.479915 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:37:38.230362 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:37:38.230327 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:37:38.442517 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:37:38.442428 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:37:59.504346 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:37:59.504316 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:37:59.570097 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:37:59.570063 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:00.363607 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:00.363569 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:00.462803 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:00.462765 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:00.970357 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:00.970318 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:01.434525 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:01.434486 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:01.536768 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:01.536733 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:06.156326 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:06.156284 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:08.754416 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:08.754377 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:09.244511 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:09.244429 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:11.529109 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:11.529070 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:11.689242 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:11.689201 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:11.773397 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:11.773366 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:13.359856 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:13.359816 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:13.741596 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:13.741479 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:13.847354 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:13.847322 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:15.197393 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:15.197360 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:16.626979 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:16.626938 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:19.394667 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:19.394628 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:21.185248 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:21.185212 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:21.294146 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:21.294109 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:24.163560 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:24.163529 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:24.269010 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:24.268975 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:25.727060 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:25.727010 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:25.998521 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:25.998482 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:29.362026 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:29.361991 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:29.465608 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:29.465577 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:29.563106 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:29.563024 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:29.634372 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:29.634280 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:32.346806 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:32.346774 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:32.446629 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:32.446588 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:35.320217 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:35.320181 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574"] Jul 05 19:38:35.323507 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:35.323487 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" Jul 05 19:38:35.327434 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:35.327410 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wk9wk\"" Jul 05 19:38:35.327605 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:35.327591 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 05 19:38:35.327712 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:35.327696 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 05 19:38:35.341128 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:35.341095 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574"] Jul 05 19:38:35.444137 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:35.444099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdfnn\" (UniqueName: \"kubernetes.io/projected/8bf0f137-f495-44ce-a01d-1bd5d8c82758-kube-api-access-vdfnn\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-d7574\" (UID: \"8bf0f137-f495-44ce-a01d-1bd5d8c82758\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" Jul 05 19:38:35.444317 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:35.444153 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8bf0f137-f495-44ce-a01d-1bd5d8c82758-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-d7574\" (UID: \"8bf0f137-f495-44ce-a01d-1bd5d8c82758\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" Jul 05 19:38:35.544819 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:35.544775 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdfnn\" (UniqueName: \"kubernetes.io/projected/8bf0f137-f495-44ce-a01d-1bd5d8c82758-kube-api-access-vdfnn\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-d7574\" (UID: \"8bf0f137-f495-44ce-a01d-1bd5d8c82758\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" Jul 05 19:38:35.544997 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:35.544835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8bf0f137-f495-44ce-a01d-1bd5d8c82758-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-d7574\" (UID: \"8bf0f137-f495-44ce-a01d-1bd5d8c82758\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" Jul 05 19:38:35.545299 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:35.545281 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8bf0f137-f495-44ce-a01d-1bd5d8c82758-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-d7574\" (UID: \"8bf0f137-f495-44ce-a01d-1bd5d8c82758\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" Jul 05 19:38:35.553960 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:35.553930 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdfnn\" (UniqueName: \"kubernetes.io/projected/8bf0f137-f495-44ce-a01d-1bd5d8c82758-kube-api-access-vdfnn\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-d7574\" (UID: \"8bf0f137-f495-44ce-a01d-1bd5d8c82758\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" Jul 05 19:38:35.634167 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:35.634071 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" Jul 05 19:38:35.794547 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:35.794520 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574"] Jul 05 19:38:35.796820 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:38:35.796796 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf0f137_f495_44ce_a01d_1bd5d8c82758.slice/crio-07ba862c1e967be66b7282f0a8ee028215255506cc86f0b1377e398e588d60cd WatchSource:0}: Error finding container 07ba862c1e967be66b7282f0a8ee028215255506cc86f0b1377e398e588d60cd: Status 404 returned error can't find the container with id 07ba862c1e967be66b7282f0a8ee028215255506cc86f0b1377e398e588d60cd Jul 05 19:38:35.908032 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:35.907937 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" event={"ID":"8bf0f137-f495-44ce-a01d-1bd5d8c82758","Type":"ContainerStarted","Data":"07ba862c1e967be66b7282f0a8ee028215255506cc86f0b1377e398e588d60cd"} Jul 05 19:38:36.057126 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:36.057088 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574"] Jul 05 19:38:36.065051 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:36.065004 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574"] Jul 05 19:38:36.106995 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:36.106949 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd"] Jul 05 19:38:36.111625 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:36.111601 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" Jul 05 19:38:36.126535 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:36.126505 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd"] Jul 05 19:38:36.149075 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:36.149020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx7fj\" (UniqueName: \"kubernetes.io/projected/10262ce3-1c86-4c47-a62f-2576070caacc-kube-api-access-jx7fj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dpmtd\" (UID: \"10262ce3-1c86-4c47-a62f-2576070caacc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" Jul 05 19:38:36.149258 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:36.149153 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10262ce3-1c86-4c47-a62f-2576070caacc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dpmtd\" (UID: \"10262ce3-1c86-4c47-a62f-2576070caacc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" Jul 05 19:38:36.250035 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:36.250001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7fj\" (UniqueName: \"kubernetes.io/projected/10262ce3-1c86-4c47-a62f-2576070caacc-kube-api-access-jx7fj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dpmtd\" (UID: \"10262ce3-1c86-4c47-a62f-2576070caacc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" Jul 05 19:38:36.250035 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:36.250082 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10262ce3-1c86-4c47-a62f-2576070caacc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dpmtd\" (UID: \"10262ce3-1c86-4c47-a62f-2576070caacc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" Jul 05 19:38:36.250424 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:36.250394 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10262ce3-1c86-4c47-a62f-2576070caacc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dpmtd\" (UID: \"10262ce3-1c86-4c47-a62f-2576070caacc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" Jul 05 19:38:36.259531 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:36.259502 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7fj\" (UniqueName: \"kubernetes.io/projected/10262ce3-1c86-4c47-a62f-2576070caacc-kube-api-access-jx7fj\") pod \"kuadrant-operator-controller-manager-8684c9ff49-dpmtd\" (UID: \"10262ce3-1c86-4c47-a62f-2576070caacc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" Jul 05 19:38:36.423649 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:36.423613 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" Jul 05 19:38:37.130573 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:37.130541 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd"] Jul 05 19:38:37.132470 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:38:37.132437 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10262ce3_1c86_4c47_a62f_2576070caacc.slice/crio-7662e50fc1987e4a8800c2d4f0bbee7bf303a8ae05c179da7abee17839885364 WatchSource:0}: Error finding container 7662e50fc1987e4a8800c2d4f0bbee7bf303a8ae05c179da7abee17839885364: Status 404 returned error can't find the container with id 7662e50fc1987e4a8800c2d4f0bbee7bf303a8ae05c179da7abee17839885364 Jul 05 19:38:37.922240 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:37.922190 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" event={"ID":"10262ce3-1c86-4c47-a62f-2576070caacc","Type":"ContainerStarted","Data":"7662e50fc1987e4a8800c2d4f0bbee7bf303a8ae05c179da7abee17839885364"} Jul 05 19:38:38.971390 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:38.971352 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:39.074256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:39.074216 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:40.940323 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:40.940288 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" podUID="8bf0f137-f495-44ce-a01d-1bd5d8c82758" containerName="manager" containerID="cri-o://79dc5822507b52f64586b98a6bcc5990256c1d2a88399c08320790e74c63d75e" gracePeriod=2 Jul 05 19:38:40.942076 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:40.942033 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" event={"ID":"10262ce3-1c86-4c47-a62f-2576070caacc","Type":"ContainerStarted","Data":"f86b8a10c43d940dc6137a0804b495b01f0a12116d1489c41f4573c741ea4e24"} Jul 05 19:38:40.942214 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:40.942189 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" Jul 05 19:38:40.943148 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:38:40.943126 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d7574\" is forbidden: User \"system:node:ip-10-0-138-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-40.ec2.internal' and this object" podUID="8bf0f137-f495-44ce-a01d-1bd5d8c82758" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" Jul 05 19:38:40.970091 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:38:40.970028 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d7574\" is forbidden: User \"system:node:ip-10-0-138-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-40.ec2.internal' and this object" podUID="8bf0f137-f495-44ce-a01d-1bd5d8c82758" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" Jul 05 19:38:40.970559 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:40.970520 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" podStartSLOduration=2.029812976 podStartE2EDuration="4.970505703s" totalImagesPullingTime="2.940692727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:38:40.966548167 +0000 UTC m=+521.966566273" watchObservedRunningTime="2026-07-05 19:38:40.970505703 +0000 UTC m=+521.970523810" Jul 05 19:38:41.185654 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:41.185624 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" Jul 05 19:38:41.188543 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:38:41.188517 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d7574\" is forbidden: User \"system:node:ip-10-0-138-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-40.ec2.internal' and this object" podUID="8bf0f137-f495-44ce-a01d-1bd5d8c82758" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" Jul 05 19:38:41.297832 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:41.297794 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdfnn\" (UniqueName: \"kubernetes.io/projected/8bf0f137-f495-44ce-a01d-1bd5d8c82758-kube-api-access-vdfnn\") pod \"8bf0f137-f495-44ce-a01d-1bd5d8c82758\" (UID: \"8bf0f137-f495-44ce-a01d-1bd5d8c82758\") " Jul 05 19:38:41.298006 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:41.297878 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8bf0f137-f495-44ce-a01d-1bd5d8c82758-extensions-socket-volume\") pod \"8bf0f137-f495-44ce-a01d-1bd5d8c82758\" (UID: \"8bf0f137-f495-44ce-a01d-1bd5d8c82758\") " Jul 05 19:38:41.298204 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:41.298183 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf0f137-f495-44ce-a01d-1bd5d8c82758-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8bf0f137-f495-44ce-a01d-1bd5d8c82758" (UID: "8bf0f137-f495-44ce-a01d-1bd5d8c82758"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:38:41.300196 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:41.300163 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf0f137-f495-44ce-a01d-1bd5d8c82758-kube-api-access-vdfnn" (OuterVolumeSpecName: "kube-api-access-vdfnn") pod "8bf0f137-f495-44ce-a01d-1bd5d8c82758" (UID: "8bf0f137-f495-44ce-a01d-1bd5d8c82758"). InnerVolumeSpecName "kube-api-access-vdfnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:38:41.399101 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:41.399036 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdfnn\" (UniqueName: \"kubernetes.io/projected/8bf0f137-f495-44ce-a01d-1bd5d8c82758-kube-api-access-vdfnn\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:38:41.399101 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:41.399094 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8bf0f137-f495-44ce-a01d-1bd5d8c82758-extensions-socket-volume\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:38:41.469111 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:41.469081 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bf0f137-f495-44ce-a01d-1bd5d8c82758" path="/var/lib/kubelet/pods/8bf0f137-f495-44ce-a01d-1bd5d8c82758/volumes" Jul 05 19:38:41.784362 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:41.784326 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:41.920924 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:41.920886 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:41.947985 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:41.947954 2573 generic.go:356] "Generic (PLEG): container finished" podID="8bf0f137-f495-44ce-a01d-1bd5d8c82758" containerID="79dc5822507b52f64586b98a6bcc5990256c1d2a88399c08320790e74c63d75e" exitCode=2 Jul 05 19:38:41.948493 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:41.948000 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" Jul 05 19:38:41.948493 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:41.948073 2573 scope.go:117] "RemoveContainer" containerID="79dc5822507b52f64586b98a6bcc5990256c1d2a88399c08320790e74c63d75e" Jul 05 19:38:41.951805 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:38:41.951781 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d7574\" is forbidden: User \"system:node:ip-10-0-138-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-40.ec2.internal' and this object" podUID="8bf0f137-f495-44ce-a01d-1bd5d8c82758" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" Jul 05 19:38:41.956811 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:41.956796 2573 scope.go:117] "RemoveContainer" containerID="79dc5822507b52f64586b98a6bcc5990256c1d2a88399c08320790e74c63d75e" Jul 05 19:38:41.957086 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:38:41.957064 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79dc5822507b52f64586b98a6bcc5990256c1d2a88399c08320790e74c63d75e\": container with ID starting with 79dc5822507b52f64586b98a6bcc5990256c1d2a88399c08320790e74c63d75e not found: ID does not exist" containerID="79dc5822507b52f64586b98a6bcc5990256c1d2a88399c08320790e74c63d75e" Jul 05 19:38:41.957151 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:41.957094 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79dc5822507b52f64586b98a6bcc5990256c1d2a88399c08320790e74c63d75e"} err="failed to get container status \"79dc5822507b52f64586b98a6bcc5990256c1d2a88399c08320790e74c63d75e\": rpc error: code = NotFound desc = could not find container \"79dc5822507b52f64586b98a6bcc5990256c1d2a88399c08320790e74c63d75e\": container with ID starting with 79dc5822507b52f64586b98a6bcc5990256c1d2a88399c08320790e74c63d75e not found: ID does not exist" Jul 05 19:38:41.961122 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:38:41.961098 2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d7574\" is forbidden: User \"system:node:ip-10-0-138-40.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-40.ec2.internal' and this object" podUID="8bf0f137-f495-44ce-a01d-1bd5d8c82758" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d7574" Jul 05 19:38:46.590311 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:46.590274 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:46.690686 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:46.690649 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:50.189840 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:50.189803 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:50.286053 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:50.286006 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:51.950304 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:51.950274 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" Jul 05 19:38:56.773081 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:56.773023 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:56.874268 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:56.874229 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:38:59.658001 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:59.657968 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd"] Jul 05 19:38:59.658494 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:59.658202 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" podUID="10262ce3-1c86-4c47-a62f-2576070caacc" containerName="manager" containerID="cri-o://f86b8a10c43d940dc6137a0804b495b01f0a12116d1489c41f4573c741ea4e24" gracePeriod=10 Jul 05 19:38:59.907803 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:38:59.907781 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" Jul 05 19:39:00.006960 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.006929 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:00.033871 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.033794 2573 generic.go:356] "Generic (PLEG): container finished" podID="10262ce3-1c86-4c47-a62f-2576070caacc" containerID="f86b8a10c43d940dc6137a0804b495b01f0a12116d1489c41f4573c741ea4e24" exitCode=0 Jul 05 19:39:00.033871 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.033843 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" Jul 05 19:39:00.034075 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.033880 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" event={"ID":"10262ce3-1c86-4c47-a62f-2576070caacc","Type":"ContainerDied","Data":"f86b8a10c43d940dc6137a0804b495b01f0a12116d1489c41f4573c741ea4e24"} Jul 05 19:39:00.034075 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.033935 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd" event={"ID":"10262ce3-1c86-4c47-a62f-2576070caacc","Type":"ContainerDied","Data":"7662e50fc1987e4a8800c2d4f0bbee7bf303a8ae05c179da7abee17839885364"} Jul 05 19:39:00.034075 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.033960 2573 scope.go:117] "RemoveContainer" containerID="f86b8a10c43d940dc6137a0804b495b01f0a12116d1489c41f4573c741ea4e24" Jul 05 19:39:00.042855 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.042839 2573 scope.go:117] "RemoveContainer" containerID="f86b8a10c43d940dc6137a0804b495b01f0a12116d1489c41f4573c741ea4e24" Jul 05 19:39:00.043152 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:39:00.043135 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86b8a10c43d940dc6137a0804b495b01f0a12116d1489c41f4573c741ea4e24\": container with ID starting with f86b8a10c43d940dc6137a0804b495b01f0a12116d1489c41f4573c741ea4e24 not found: ID does not exist" containerID="f86b8a10c43d940dc6137a0804b495b01f0a12116d1489c41f4573c741ea4e24" Jul 05 19:39:00.043219 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.043158 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86b8a10c43d940dc6137a0804b495b01f0a12116d1489c41f4573c741ea4e24"} err="failed to get container status \"f86b8a10c43d940dc6137a0804b495b01f0a12116d1489c41f4573c741ea4e24\": rpc error: code = NotFound desc = could not find container \"f86b8a10c43d940dc6137a0804b495b01f0a12116d1489c41f4573c741ea4e24\": container with ID starting with f86b8a10c43d940dc6137a0804b495b01f0a12116d1489c41f4573c741ea4e24 not found: ID does not exist" Jul 05 19:39:00.072522 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.072483 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10262ce3-1c86-4c47-a62f-2576070caacc-extensions-socket-volume\") pod \"10262ce3-1c86-4c47-a62f-2576070caacc\" (UID: \"10262ce3-1c86-4c47-a62f-2576070caacc\") " Jul 05 19:39:00.072677 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.072550 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx7fj\" (UniqueName: \"kubernetes.io/projected/10262ce3-1c86-4c47-a62f-2576070caacc-kube-api-access-jx7fj\") pod \"10262ce3-1c86-4c47-a62f-2576070caacc\" (UID: \"10262ce3-1c86-4c47-a62f-2576070caacc\") " Jul 05 19:39:00.072888 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.072864 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10262ce3-1c86-4c47-a62f-2576070caacc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "10262ce3-1c86-4c47-a62f-2576070caacc" (UID: "10262ce3-1c86-4c47-a62f-2576070caacc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 19:39:00.074718 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.074696 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10262ce3-1c86-4c47-a62f-2576070caacc-kube-api-access-jx7fj" (OuterVolumeSpecName: "kube-api-access-jx7fj") pod "10262ce3-1c86-4c47-a62f-2576070caacc" (UID: "10262ce3-1c86-4c47-a62f-2576070caacc"). InnerVolumeSpecName "kube-api-access-jx7fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:39:00.103466 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.103426 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:00.173887 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.173852 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx7fj\" (UniqueName: \"kubernetes.io/projected/10262ce3-1c86-4c47-a62f-2576070caacc-kube-api-access-jx7fj\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:39:00.173887 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.173881 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10262ce3-1c86-4c47-a62f-2576070caacc-extensions-socket-volume\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:39:00.366301 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.366238 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd"] Jul 05 19:39:00.367827 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:00.367806 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-dpmtd"] Jul 05 19:39:01.468434 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:01.468400 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10262ce3-1c86-4c47-a62f-2576070caacc" path="/var/lib/kubelet/pods/10262ce3-1c86-4c47-a62f-2576070caacc/volumes" Jul 05 19:39:04.575910 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:04.575872 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:04.676161 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:04.676117 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:04.779527 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:04.779487 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:04.829995 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:04.829905 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:10.168168 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:10.168128 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:10.268027 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:10.267992 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:12.523005 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:12.522972 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:14.038914 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:14.038877 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:14.139903 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:14.139870 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:15.977992 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:15.977958 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:16.036926 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:16.036887 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:19.718501 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.718463 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jnkrp"] Jul 05 19:39:19.718874 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.718835 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf0f137-f495-44ce-a01d-1bd5d8c82758" containerName="manager" Jul 05 19:39:19.718874 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.718847 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf0f137-f495-44ce-a01d-1bd5d8c82758" containerName="manager" Jul 05 19:39:19.718874 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.718860 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10262ce3-1c86-4c47-a62f-2576070caacc" containerName="manager" Jul 05 19:39:19.718874 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.718868 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="10262ce3-1c86-4c47-a62f-2576070caacc" containerName="manager" Jul 05 19:39:19.719005 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.718931 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="10262ce3-1c86-4c47-a62f-2576070caacc" containerName="manager" Jul 05 19:39:19.719005 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.718940 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf0f137-f495-44ce-a01d-1bd5d8c82758" containerName="manager" Jul 05 19:39:19.720677 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.720644 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:19.721672 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.721657 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" Jul 05 19:39:19.724805 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.724772 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 05 19:39:19.725143 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.725087 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 05 19:39:19.725258 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.725240 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 05 19:39:19.726036 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.726013 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wffnq\"" Jul 05 19:39:19.735575 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.735552 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jnkrp"] Jul 05 19:39:19.743388 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.743359 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d6d0693d-0912-4dd4-b9df-2a0bf27fedc5-config-file\") pod \"limitador-limitador-64c94fd64f-jnkrp\" (UID: \"d6d0693d-0912-4dd4-b9df-2a0bf27fedc5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" Jul 05 19:39:19.743489 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.743420 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vjwq\" (UniqueName: \"kubernetes.io/projected/d6d0693d-0912-4dd4-b9df-2a0bf27fedc5-kube-api-access-4vjwq\") pod \"limitador-limitador-64c94fd64f-jnkrp\" (UID: \"d6d0693d-0912-4dd4-b9df-2a0bf27fedc5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" Jul 05 19:39:19.814852 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.814811 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jnkrp"] Jul 05 19:39:19.817754 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.817726 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:19.844143 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.844109 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d6d0693d-0912-4dd4-b9df-2a0bf27fedc5-config-file\") pod \"limitador-limitador-64c94fd64f-jnkrp\" (UID: \"d6d0693d-0912-4dd4-b9df-2a0bf27fedc5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" Jul 05 19:39:19.844296 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.844154 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vjwq\" (UniqueName: \"kubernetes.io/projected/d6d0693d-0912-4dd4-b9df-2a0bf27fedc5-kube-api-access-4vjwq\") pod \"limitador-limitador-64c94fd64f-jnkrp\" (UID: \"d6d0693d-0912-4dd4-b9df-2a0bf27fedc5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" Jul 05 19:39:19.844711 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.844693 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d6d0693d-0912-4dd4-b9df-2a0bf27fedc5-config-file\") pod \"limitador-limitador-64c94fd64f-jnkrp\" (UID: \"d6d0693d-0912-4dd4-b9df-2a0bf27fedc5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" Jul 05 19:39:19.852587 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.852564 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vjwq\" (UniqueName: \"kubernetes.io/projected/d6d0693d-0912-4dd4-b9df-2a0bf27fedc5-kube-api-access-4vjwq\") pod \"limitador-limitador-64c94fd64f-jnkrp\" (UID: \"d6d0693d-0912-4dd4-b9df-2a0bf27fedc5\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" Jul 05 19:39:19.872694 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:19.872658 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:20.034099 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.034064 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" Jul 05 19:39:20.169414 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.169388 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jnkrp"] Jul 05 19:39:20.171567 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:39:20.171535 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d0693d_0912_4dd4_b9df_2a0bf27fedc5.slice/crio-298b9dfcc9878af641d0d69b84d7b8c4bd3dfd38d212b06c710fcb2c8b2c0db2 WatchSource:0}: Error finding container 298b9dfcc9878af641d0d69b84d7b8c4bd3dfd38d212b06c710fcb2c8b2c0db2: Status 404 returned error can't find the container with id 298b9dfcc9878af641d0d69b84d7b8c4bd3dfd38d212b06c710fcb2c8b2c0db2 Jul 05 19:39:20.520884 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.520852 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-8cbbv"] Jul 05 19:39:20.523842 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.523818 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8cbbv" Jul 05 19:39:20.526507 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.526485 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5w68x\"" Jul 05 19:39:20.530809 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.530782 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8cbbv"] Jul 05 19:39:20.549024 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.548993 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:20.549397 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.549378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vxx5\" (UniqueName: \"kubernetes.io/projected/8f9b15d4-67f7-4bdf-9adf-043791c4e348-kube-api-access-9vxx5\") pod \"authorino-58cd955675-8cbbv\" (UID: \"8f9b15d4-67f7-4bdf-9adf-043791c4e348\") " pod="kuadrant-system/authorino-58cd955675-8cbbv" Jul 05 19:39:20.605983 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.605952 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:20.650253 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.650224 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vxx5\" (UniqueName: \"kubernetes.io/projected/8f9b15d4-67f7-4bdf-9adf-043791c4e348-kube-api-access-9vxx5\") pod \"authorino-58cd955675-8cbbv\" (UID: \"8f9b15d4-67f7-4bdf-9adf-043791c4e348\") " pod="kuadrant-system/authorino-58cd955675-8cbbv" Jul 05 19:39:20.662909 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.662881 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vxx5\" (UniqueName: \"kubernetes.io/projected/8f9b15d4-67f7-4bdf-9adf-043791c4e348-kube-api-access-9vxx5\") pod \"authorino-58cd955675-8cbbv\" (UID: \"8f9b15d4-67f7-4bdf-9adf-043791c4e348\") " pod="kuadrant-system/authorino-58cd955675-8cbbv" Jul 05 19:39:20.684057 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.684013 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-76n9m"] Jul 05 19:39:20.686247 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.686227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-76n9m" Jul 05 19:39:20.694235 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.694214 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-76n9m"] Jul 05 19:39:20.750849 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.750811 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nz79\" (UniqueName: \"kubernetes.io/projected/0989624d-8659-40ff-90c3-03ade136a506-kube-api-access-8nz79\") pod \"authorino-75d556bd77-76n9m\" (UID: \"0989624d-8659-40ff-90c3-03ade136a506\") " pod="kuadrant-system/authorino-75d556bd77-76n9m" Jul 05 19:39:20.843551 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.843459 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8cbbv" Jul 05 19:39:20.852144 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.852110 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nz79\" (UniqueName: \"kubernetes.io/projected/0989624d-8659-40ff-90c3-03ade136a506-kube-api-access-8nz79\") pod \"authorino-75d556bd77-76n9m\" (UID: \"0989624d-8659-40ff-90c3-03ade136a506\") " pod="kuadrant-system/authorino-75d556bd77-76n9m" Jul 05 19:39:20.862648 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.862606 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nz79\" (UniqueName: \"kubernetes.io/projected/0989624d-8659-40ff-90c3-03ade136a506-kube-api-access-8nz79\") pod \"authorino-75d556bd77-76n9m\" (UID: \"0989624d-8659-40ff-90c3-03ade136a506\") " pod="kuadrant-system/authorino-75d556bd77-76n9m" Jul 05 19:39:20.996161 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:20.996123 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-76n9m" Jul 05 19:39:21.030957 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:21.030900 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8cbbv"] Jul 05 19:39:21.040175 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:39:21.038602 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f9b15d4_67f7_4bdf_9adf_043791c4e348.slice/crio-ce20066c78d645e2f1e33582bb2a057cbbbcf152b33c5a89bb4daa830a9dcd20 WatchSource:0}: Error finding container ce20066c78d645e2f1e33582bb2a057cbbbcf152b33c5a89bb4daa830a9dcd20: Status 404 returned error can't find the container with id ce20066c78d645e2f1e33582bb2a057cbbbcf152b33c5a89bb4daa830a9dcd20 Jul 05 19:39:21.137033 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:21.136993 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8cbbv" event={"ID":"8f9b15d4-67f7-4bdf-9adf-043791c4e348","Type":"ContainerStarted","Data":"ce20066c78d645e2f1e33582bb2a057cbbbcf152b33c5a89bb4daa830a9dcd20"} Jul 05 19:39:21.139317 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:21.139245 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" event={"ID":"d6d0693d-0912-4dd4-b9df-2a0bf27fedc5","Type":"ContainerStarted","Data":"298b9dfcc9878af641d0d69b84d7b8c4bd3dfd38d212b06c710fcb2c8b2c0db2"} Jul 05 19:39:21.169379 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:21.169242 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-76n9m"] Jul 05 19:39:21.171186 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:39:21.171155 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0989624d_8659_40ff_90c3_03ade136a506.slice/crio-5374c26c3f3691fca34580ca79aa7d2379e99a1fbcccc263d1376e4bc19106d2 WatchSource:0}: Error finding container 5374c26c3f3691fca34580ca79aa7d2379e99a1fbcccc263d1376e4bc19106d2: Status 404 returned error can't find the container with id 5374c26c3f3691fca34580ca79aa7d2379e99a1fbcccc263d1376e4bc19106d2 Jul 05 19:39:22.148794 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:22.148678 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-76n9m" event={"ID":"0989624d-8659-40ff-90c3-03ade136a506","Type":"ContainerStarted","Data":"5374c26c3f3691fca34580ca79aa7d2379e99a1fbcccc263d1376e4bc19106d2"} Jul 05 19:39:24.034810 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:24.034774 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:25.168007 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:25.167895 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-76n9m" event={"ID":"0989624d-8659-40ff-90c3-03ade136a506","Type":"ContainerStarted","Data":"0288ee4671f27a396a7c17f517556bcade0cb0a0da68ebd33ced20554af26492"} Jul 05 19:39:25.169358 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:25.169335 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8cbbv" event={"ID":"8f9b15d4-67f7-4bdf-9adf-043791c4e348","Type":"ContainerStarted","Data":"79d7b9dd4d4ed690781573bbb05983c9b1b917dc666cc9aa0540c9d83ae979f2"} Jul 05 19:39:25.170606 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:25.170585 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" event={"ID":"d6d0693d-0912-4dd4-b9df-2a0bf27fedc5","Type":"ContainerStarted","Data":"dd1bc0ddbbb6ca85d944d3c62313ab9f7ab897334cdd89f845bb440008ecb0d2"} Jul 05 19:39:25.170736 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:25.170722 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" Jul 05 19:39:25.188823 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:25.188776 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-76n9m" podStartSLOduration=1.480375339 podStartE2EDuration="5.188764379s" totalImagesPullingTime="3.70838904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:39:25.187941913 +0000 UTC m=+566.187965845" watchObservedRunningTime="2026-07-05 19:39:25.188764379 +0000 UTC m=+566.188782486" Jul 05 19:39:25.205971 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:25.205916 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-8cbbv" podStartSLOduration=1.354359525 podStartE2EDuration="5.205904056s" totalImagesPullingTime="3.851544531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:39:25.204352372 +0000 UTC m=+566.204370478" watchObservedRunningTime="2026-07-05 19:39:25.205904056 +0000 UTC m=+566.205922206" Jul 05 19:39:25.213877 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:25.213846 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8cbbv"] Jul 05 19:39:25.222584 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:25.222537 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" podStartSLOduration=1.552139299 podStartE2EDuration="6.222515397s" totalImagesPullingTime="4.670376098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:39:25.2212834 +0000 UTC m=+566.221301507" watchObservedRunningTime="2026-07-05 19:39:25.222515397 +0000 UTC m=+566.222533531" Jul 05 19:39:27.180231 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:27.180166 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-8cbbv" podUID="8f9b15d4-67f7-4bdf-9adf-043791c4e348" containerName="authorino" containerID="cri-o://79d7b9dd4d4ed690781573bbb05983c9b1b917dc666cc9aa0540c9d83ae979f2" gracePeriod=30 Jul 05 19:39:27.424900 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:27.424877 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8cbbv" Jul 05 19:39:27.517388 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:27.517364 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vxx5\" (UniqueName: \"kubernetes.io/projected/8f9b15d4-67f7-4bdf-9adf-043791c4e348-kube-api-access-9vxx5\") pod \"8f9b15d4-67f7-4bdf-9adf-043791c4e348\" (UID: \"8f9b15d4-67f7-4bdf-9adf-043791c4e348\") " Jul 05 19:39:27.519553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:27.519529 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9b15d4-67f7-4bdf-9adf-043791c4e348-kube-api-access-9vxx5" (OuterVolumeSpecName: "kube-api-access-9vxx5") pod "8f9b15d4-67f7-4bdf-9adf-043791c4e348" (UID: "8f9b15d4-67f7-4bdf-9adf-043791c4e348"). InnerVolumeSpecName "kube-api-access-9vxx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:39:27.618507 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:27.618475 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vxx5\" (UniqueName: \"kubernetes.io/projected/8f9b15d4-67f7-4bdf-9adf-043791c4e348-kube-api-access-9vxx5\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:39:28.186032 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:28.185996 2573 generic.go:356] "Generic (PLEG): container finished" podID="8f9b15d4-67f7-4bdf-9adf-043791c4e348" containerID="79d7b9dd4d4ed690781573bbb05983c9b1b917dc666cc9aa0540c9d83ae979f2" exitCode=0 Jul 05 19:39:28.186493 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:28.186033 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8cbbv" Jul 05 19:39:28.186493 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:28.186096 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8cbbv" event={"ID":"8f9b15d4-67f7-4bdf-9adf-043791c4e348","Type":"ContainerDied","Data":"79d7b9dd4d4ed690781573bbb05983c9b1b917dc666cc9aa0540c9d83ae979f2"} Jul 05 19:39:28.186493 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:28.186145 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8cbbv" event={"ID":"8f9b15d4-67f7-4bdf-9adf-043791c4e348","Type":"ContainerDied","Data":"ce20066c78d645e2f1e33582bb2a057cbbbcf152b33c5a89bb4daa830a9dcd20"} Jul 05 19:39:28.186493 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:28.186168 2573 scope.go:117] "RemoveContainer" containerID="79d7b9dd4d4ed690781573bbb05983c9b1b917dc666cc9aa0540c9d83ae979f2" Jul 05 19:39:28.195267 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:28.195252 2573 scope.go:117] "RemoveContainer" containerID="79d7b9dd4d4ed690781573bbb05983c9b1b917dc666cc9aa0540c9d83ae979f2" Jul 05 19:39:28.195548 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:39:28.195528 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d7b9dd4d4ed690781573bbb05983c9b1b917dc666cc9aa0540c9d83ae979f2\": container with ID starting with 79d7b9dd4d4ed690781573bbb05983c9b1b917dc666cc9aa0540c9d83ae979f2 not found: ID does not exist" containerID="79d7b9dd4d4ed690781573bbb05983c9b1b917dc666cc9aa0540c9d83ae979f2" Jul 05 19:39:28.195609 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:28.195554 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d7b9dd4d4ed690781573bbb05983c9b1b917dc666cc9aa0540c9d83ae979f2"} err="failed to get container status \"79d7b9dd4d4ed690781573bbb05983c9b1b917dc666cc9aa0540c9d83ae979f2\": rpc error: code = NotFound desc = could not find container \"79d7b9dd4d4ed690781573bbb05983c9b1b917dc666cc9aa0540c9d83ae979f2\": container with ID starting with 79d7b9dd4d4ed690781573bbb05983c9b1b917dc666cc9aa0540c9d83ae979f2 not found: ID does not exist" Jul 05 19:39:28.207111 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:28.207086 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8cbbv"] Jul 05 19:39:28.210438 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:28.210407 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-8cbbv"] Jul 05 19:39:29.468840 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:29.468803 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f9b15d4-67f7-4bdf-9adf-043791c4e348" path="/var/lib/kubelet/pods/8f9b15d4-67f7-4bdf-9adf-043791c4e348/volumes" Jul 05 19:39:31.002708 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:31.002675 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:31.049997 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:31.049966 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:34.854657 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:34.854622 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jnkrp"] Jul 05 19:39:34.855070 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:34.854828 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" podUID="d6d0693d-0912-4dd4-b9df-2a0bf27fedc5" containerName="limitador" containerID="cri-o://dd1bc0ddbbb6ca85d944d3c62313ab9f7ab897334cdd89f845bb440008ecb0d2" gracePeriod=30 Jul 05 19:39:34.855642 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:34.855622 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" Jul 05 19:39:35.221234 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:35.221204 2573 generic.go:356] "Generic (PLEG): container finished" podID="d6d0693d-0912-4dd4-b9df-2a0bf27fedc5" containerID="dd1bc0ddbbb6ca85d944d3c62313ab9f7ab897334cdd89f845bb440008ecb0d2" exitCode=0 Jul 05 19:39:35.221398 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:35.221273 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" event={"ID":"d6d0693d-0912-4dd4-b9df-2a0bf27fedc5","Type":"ContainerDied","Data":"dd1bc0ddbbb6ca85d944d3c62313ab9f7ab897334cdd89f845bb440008ecb0d2"} Jul 05 19:39:35.800492 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:35.800470 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" Jul 05 19:39:35.887488 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:35.887458 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d6d0693d-0912-4dd4-b9df-2a0bf27fedc5-config-file\") pod \"d6d0693d-0912-4dd4-b9df-2a0bf27fedc5\" (UID: \"d6d0693d-0912-4dd4-b9df-2a0bf27fedc5\") " Jul 05 19:39:35.887885 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:35.887517 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vjwq\" (UniqueName: \"kubernetes.io/projected/d6d0693d-0912-4dd4-b9df-2a0bf27fedc5-kube-api-access-4vjwq\") pod \"d6d0693d-0912-4dd4-b9df-2a0bf27fedc5\" (UID: \"d6d0693d-0912-4dd4-b9df-2a0bf27fedc5\") " Jul 05 19:39:35.887885 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:35.887818 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d0693d-0912-4dd4-b9df-2a0bf27fedc5-config-file" (OuterVolumeSpecName: "config-file") pod "d6d0693d-0912-4dd4-b9df-2a0bf27fedc5" (UID: "d6d0693d-0912-4dd4-b9df-2a0bf27fedc5"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:39:35.889751 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:35.889729 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d0693d-0912-4dd4-b9df-2a0bf27fedc5-kube-api-access-4vjwq" (OuterVolumeSpecName: "kube-api-access-4vjwq") pod "d6d0693d-0912-4dd4-b9df-2a0bf27fedc5" (UID: "d6d0693d-0912-4dd4-b9df-2a0bf27fedc5"). InnerVolumeSpecName "kube-api-access-4vjwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:39:35.988919 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:35.988829 2573 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d6d0693d-0912-4dd4-b9df-2a0bf27fedc5-config-file\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:39:35.988919 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:35.988862 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vjwq\" (UniqueName: \"kubernetes.io/projected/d6d0693d-0912-4dd4-b9df-2a0bf27fedc5-kube-api-access-4vjwq\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:39:36.226888 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:36.226860 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" Jul 05 19:39:36.227080 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:36.226906 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jnkrp" event={"ID":"d6d0693d-0912-4dd4-b9df-2a0bf27fedc5","Type":"ContainerDied","Data":"298b9dfcc9878af641d0d69b84d7b8c4bd3dfd38d212b06c710fcb2c8b2c0db2"} Jul 05 19:39:36.227080 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:36.226967 2573 scope.go:117] "RemoveContainer" containerID="dd1bc0ddbbb6ca85d944d3c62313ab9f7ab897334cdd89f845bb440008ecb0d2" Jul 05 19:39:36.248980 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:36.248513 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jnkrp"] Jul 05 19:39:36.252825 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:36.252785 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jnkrp"] Jul 05 19:39:37.469197 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:37.469164 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6d0693d-0912-4dd4-b9df-2a0bf27fedc5" path="/var/lib/kubelet/pods/d6d0693d-0912-4dd4-b9df-2a0bf27fedc5/volumes" Jul 05 19:39:39.548679 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.548646 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-r78zp"] Jul 05 19:39:39.549034 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.548997 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6d0693d-0912-4dd4-b9df-2a0bf27fedc5" containerName="limitador" Jul 05 19:39:39.549034 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.549007 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6d0693d-0912-4dd4-b9df-2a0bf27fedc5" containerName="limitador" Jul 05 19:39:39.549034 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.549023 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9b15d4-67f7-4bdf-9adf-043791c4e348" containerName="authorino" Jul 05 19:39:39.549034 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.549028 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9b15d4-67f7-4bdf-9adf-043791c4e348" containerName="authorino" Jul 05 19:39:39.549196 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.549106 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6d0693d-0912-4dd4-b9df-2a0bf27fedc5" containerName="limitador" Jul 05 19:39:39.549196 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.549115 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9b15d4-67f7-4bdf-9adf-043791c4e348" containerName="authorino" Jul 05 19:39:39.551793 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.551777 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-r78zp" Jul 05 19:39:39.554360 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.554337 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v2wjl\"" Jul 05 19:39:39.554489 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.554362 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 05 19:39:39.559076 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.558889 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-r78zp"] Jul 05 19:39:39.575679 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.575646 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:39.620989 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.620957 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmlz\" (UniqueName: \"kubernetes.io/projected/5523224c-838f-4e26-b02f-802e67632aaa-kube-api-access-gqmlz\") pod \"postgres-69fb4594cf-r78zp\" (UID: \"5523224c-838f-4e26-b02f-802e67632aaa\") " pod="opendatahub/postgres-69fb4594cf-r78zp" Jul 05 19:39:39.621463 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.621436 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5523224c-838f-4e26-b02f-802e67632aaa-data\") pod \"postgres-69fb4594cf-r78zp\" (UID: \"5523224c-838f-4e26-b02f-802e67632aaa\") " pod="opendatahub/postgres-69fb4594cf-r78zp" Jul 05 19:39:39.722876 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.722839 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmlz\" (UniqueName: \"kubernetes.io/projected/5523224c-838f-4e26-b02f-802e67632aaa-kube-api-access-gqmlz\") pod \"postgres-69fb4594cf-r78zp\" (UID: \"5523224c-838f-4e26-b02f-802e67632aaa\") " pod="opendatahub/postgres-69fb4594cf-r78zp" Jul 05 19:39:39.723089 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.722899 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5523224c-838f-4e26-b02f-802e67632aaa-data\") pod \"postgres-69fb4594cf-r78zp\" (UID: \"5523224c-838f-4e26-b02f-802e67632aaa\") " pod="opendatahub/postgres-69fb4594cf-r78zp" Jul 05 19:39:39.723308 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.723287 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5523224c-838f-4e26-b02f-802e67632aaa-data\") pod \"postgres-69fb4594cf-r78zp\" (UID: \"5523224c-838f-4e26-b02f-802e67632aaa\") " pod="opendatahub/postgres-69fb4594cf-r78zp" Jul 05 19:39:39.730877 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.730855 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmlz\" (UniqueName: \"kubernetes.io/projected/5523224c-838f-4e26-b02f-802e67632aaa-kube-api-access-gqmlz\") pod \"postgres-69fb4594cf-r78zp\" (UID: \"5523224c-838f-4e26-b02f-802e67632aaa\") " pod="opendatahub/postgres-69fb4594cf-r78zp" Jul 05 19:39:39.864296 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.864208 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-r78zp" Jul 05 19:39:39.994482 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:39.994459 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-r78zp"] Jul 05 19:39:39.996303 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:39:39.996277 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5523224c_838f_4e26_b02f_802e67632aaa.slice/crio-1cf9c607211b0d4c5f3516c31b6fa797463ae19adcb50a8076814f5871e5c0f8 WatchSource:0}: Error finding container 1cf9c607211b0d4c5f3516c31b6fa797463ae19adcb50a8076814f5871e5c0f8: Status 404 returned error can't find the container with id 1cf9c607211b0d4c5f3516c31b6fa797463ae19adcb50a8076814f5871e5c0f8 Jul 05 19:39:40.247153 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:40.247033 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-r78zp" event={"ID":"5523224c-838f-4e26-b02f-802e67632aaa","Type":"ContainerStarted","Data":"1cf9c607211b0d4c5f3516c31b6fa797463ae19adcb50a8076814f5871e5c0f8"} Jul 05 19:39:45.276304 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:45.276263 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-r78zp" event={"ID":"5523224c-838f-4e26-b02f-802e67632aaa","Type":"ContainerStarted","Data":"f72f54a82268c9805c1711668c7c51604bd9b94fb62c67b10b120d39efbe56d8"} Jul 05 19:39:45.276819 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:45.276387 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-r78zp" Jul 05 19:39:45.293757 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:45.293706 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-r78zp" podStartSLOduration=1.485844414 podStartE2EDuration="6.293690082s" totalImagesPullingTime="4.807845668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:39:45.291809123 +0000 UTC m=+586.291827230" watchObservedRunningTime="2026-07-05 19:39:45.293690082 +0000 UTC m=+586.293708188" Jul 05 19:39:51.312093 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:51.312063 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-r78zp" Jul 05 19:39:51.616395 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:51.616298 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:51.846081 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:51.846026 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6hfsl"] Jul 05 19:39:51.978251 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:51.978160 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6hfsl"] Jul 05 19:39:51.978406 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:51.978291 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6hfsl" Jul 05 19:39:52.083466 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.083434 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6hfsl"] Jul 05 19:39:52.083746 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:39:52.083724 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-kwf8m], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-6hfsl" podUID="18df80de-1729-43c8-8ffe-1643f315ab6d" Jul 05 19:39:52.115289 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.115256 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c4db5bbd-59d5g"] Jul 05 19:39:52.126941 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.126913 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4db5bbd-59d5g" Jul 05 19:39:52.127093 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.127020 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c4db5bbd-59d5g"] Jul 05 19:39:52.133944 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.133893 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwf8m\" (UniqueName: \"kubernetes.io/projected/18df80de-1729-43c8-8ffe-1643f315ab6d-kube-api-access-kwf8m\") pod \"authorino-5d4b67c446-6hfsl\" (UID: \"18df80de-1729-43c8-8ffe-1643f315ab6d\") " pod="kuadrant-system/authorino-5d4b67c446-6hfsl" Jul 05 19:39:52.148886 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.148857 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:52.234798 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.234710 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwf8m\" (UniqueName: \"kubernetes.io/projected/18df80de-1729-43c8-8ffe-1643f315ab6d-kube-api-access-kwf8m\") pod \"authorino-5d4b67c446-6hfsl\" (UID: \"18df80de-1729-43c8-8ffe-1643f315ab6d\") " pod="kuadrant-system/authorino-5d4b67c446-6hfsl" Jul 05 19:39:52.234798 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.234755 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q56z\" (UniqueName: \"kubernetes.io/projected/abff6727-9eea-4d29-88b8-694b70e10e0a-kube-api-access-4q56z\") pod \"authorino-6c4db5bbd-59d5g\" (UID: \"abff6727-9eea-4d29-88b8-694b70e10e0a\") " pod="kuadrant-system/authorino-6c4db5bbd-59d5g" Jul 05 19:39:52.242985 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.242962 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwf8m\" (UniqueName: \"kubernetes.io/projected/18df80de-1729-43c8-8ffe-1643f315ab6d-kube-api-access-kwf8m\") pod \"authorino-5d4b67c446-6hfsl\" (UID: \"18df80de-1729-43c8-8ffe-1643f315ab6d\") " pod="kuadrant-system/authorino-5d4b67c446-6hfsl" Jul 05 19:39:52.309468 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.309440 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6hfsl" Jul 05 19:39:52.316268 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.316245 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6hfsl" Jul 05 19:39:52.336272 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.336242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q56z\" (UniqueName: \"kubernetes.io/projected/abff6727-9eea-4d29-88b8-694b70e10e0a-kube-api-access-4q56z\") pod \"authorino-6c4db5bbd-59d5g\" (UID: \"abff6727-9eea-4d29-88b8-694b70e10e0a\") " pod="kuadrant-system/authorino-6c4db5bbd-59d5g" Jul 05 19:39:52.344263 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.344238 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q56z\" (UniqueName: \"kubernetes.io/projected/abff6727-9eea-4d29-88b8-694b70e10e0a-kube-api-access-4q56z\") pod \"authorino-6c4db5bbd-59d5g\" (UID: \"abff6727-9eea-4d29-88b8-694b70e10e0a\") " pod="kuadrant-system/authorino-6c4db5bbd-59d5g" Jul 05 19:39:52.407349 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.407314 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c4db5bbd-59d5g"] Jul 05 19:39:52.407611 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.407599 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4db5bbd-59d5g" Jul 05 19:39:52.436703 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.436678 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwf8m\" (UniqueName: \"kubernetes.io/projected/18df80de-1729-43c8-8ffe-1643f315ab6d-kube-api-access-kwf8m\") pod \"18df80de-1729-43c8-8ffe-1643f315ab6d\" (UID: \"18df80de-1729-43c8-8ffe-1643f315ab6d\") " Jul 05 19:39:52.437855 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.437827 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78b8656c68-zhw78"] Jul 05 19:39:52.439083 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.439028 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18df80de-1729-43c8-8ffe-1643f315ab6d-kube-api-access-kwf8m" (OuterVolumeSpecName: "kube-api-access-kwf8m") pod "18df80de-1729-43c8-8ffe-1643f315ab6d" (UID: "18df80de-1729-43c8-8ffe-1643f315ab6d"). InnerVolumeSpecName "kube-api-access-kwf8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:39:52.440824 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.440802 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78b8656c68-zhw78" Jul 05 19:39:52.444186 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.444056 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 05 19:39:52.450531 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.450496 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78b8656c68-zhw78"] Jul 05 19:39:52.538379 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.538339 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w68nh\" (UniqueName: \"kubernetes.io/projected/e2850f87-838b-48a5-a7fd-d0790268e82e-kube-api-access-w68nh\") pod \"authorino-78b8656c68-zhw78\" (UID: \"e2850f87-838b-48a5-a7fd-d0790268e82e\") " pod="kuadrant-system/authorino-78b8656c68-zhw78" Jul 05 19:39:52.538861 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.538747 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2850f87-838b-48a5-a7fd-d0790268e82e-tls-cert\") pod \"authorino-78b8656c68-zhw78\" (UID: \"e2850f87-838b-48a5-a7fd-d0790268e82e\") " pod="kuadrant-system/authorino-78b8656c68-zhw78" Jul 05 19:39:52.538964 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.538910 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwf8m\" (UniqueName: \"kubernetes.io/projected/18df80de-1729-43c8-8ffe-1643f315ab6d-kube-api-access-kwf8m\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:39:52.547204 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.547177 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c4db5bbd-59d5g"] Jul 05 19:39:52.548961 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:39:52.548933 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabff6727_9eea_4d29_88b8_694b70e10e0a.slice/crio-be2966458121eb25d63d6a86a0ff7b2fe02a72f8b3cf66ddb372d03cafc59131 WatchSource:0}: Error finding container be2966458121eb25d63d6a86a0ff7b2fe02a72f8b3cf66ddb372d03cafc59131: Status 404 returned error can't find the container with id be2966458121eb25d63d6a86a0ff7b2fe02a72f8b3cf66ddb372d03cafc59131 Jul 05 19:39:52.640329 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.640281 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2850f87-838b-48a5-a7fd-d0790268e82e-tls-cert\") pod \"authorino-78b8656c68-zhw78\" (UID: \"e2850f87-838b-48a5-a7fd-d0790268e82e\") " pod="kuadrant-system/authorino-78b8656c68-zhw78" Jul 05 19:39:52.640518 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.640358 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68nh\" (UniqueName: \"kubernetes.io/projected/e2850f87-838b-48a5-a7fd-d0790268e82e-kube-api-access-w68nh\") pod \"authorino-78b8656c68-zhw78\" (UID: \"e2850f87-838b-48a5-a7fd-d0790268e82e\") " pod="kuadrant-system/authorino-78b8656c68-zhw78" Jul 05 19:39:52.642903 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.642883 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2850f87-838b-48a5-a7fd-d0790268e82e-tls-cert\") pod \"authorino-78b8656c68-zhw78\" (UID: \"e2850f87-838b-48a5-a7fd-d0790268e82e\") " pod="kuadrant-system/authorino-78b8656c68-zhw78" Jul 05 19:39:52.649178 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.649158 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w68nh\" (UniqueName: \"kubernetes.io/projected/e2850f87-838b-48a5-a7fd-d0790268e82e-kube-api-access-w68nh\") pod \"authorino-78b8656c68-zhw78\" (UID: \"e2850f87-838b-48a5-a7fd-d0790268e82e\") " pod="kuadrant-system/authorino-78b8656c68-zhw78" Jul 05 19:39:52.755840 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.755809 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78b8656c68-zhw78" Jul 05 19:39:52.887024 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:52.886998 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78b8656c68-zhw78"] Jul 05 19:39:52.903243 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:39:52.903204 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2850f87_838b_48a5_a7fd_d0790268e82e.slice/crio-acbfa4a3b611a6fd9391bd591a3d6c7876aba92490c01121cc9ac3304190f0fa WatchSource:0}: Error finding container acbfa4a3b611a6fd9391bd591a3d6c7876aba92490c01121cc9ac3304190f0fa: Status 404 returned error can't find the container with id acbfa4a3b611a6fd9391bd591a3d6c7876aba92490c01121cc9ac3304190f0fa Jul 05 19:39:53.318416 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:53.318386 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78b8656c68-zhw78" event={"ID":"e2850f87-838b-48a5-a7fd-d0790268e82e","Type":"ContainerStarted","Data":"acbfa4a3b611a6fd9391bd591a3d6c7876aba92490c01121cc9ac3304190f0fa"} Jul 05 19:39:53.320227 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:53.320200 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c4db5bbd-59d5g" event={"ID":"abff6727-9eea-4d29-88b8-694b70e10e0a","Type":"ContainerStarted","Data":"fd91f232fa5b1e859c5f535689fdab48176e500fe6a913b06b85302d586cfff2"} Jul 05 19:39:53.320227 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:53.320212 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-6c4db5bbd-59d5g" podUID="abff6727-9eea-4d29-88b8-694b70e10e0a" containerName="authorino" containerID="cri-o://fd91f232fa5b1e859c5f535689fdab48176e500fe6a913b06b85302d586cfff2" gracePeriod=30 Jul 05 19:39:53.320389 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:53.320231 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6hfsl" Jul 05 19:39:53.320389 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:53.320244 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c4db5bbd-59d5g" event={"ID":"abff6727-9eea-4d29-88b8-694b70e10e0a","Type":"ContainerStarted","Data":"be2966458121eb25d63d6a86a0ff7b2fe02a72f8b3cf66ddb372d03cafc59131"} Jul 05 19:39:53.334375 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:53.334332 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c4db5bbd-59d5g" podStartSLOduration=1.015862026 podStartE2EDuration="1.334315975s" totalImagesPullingTime="318.453949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:39:53.333571588 +0000 UTC m=+594.333589692" watchObservedRunningTime="2026-07-05 19:39:53.334315975 +0000 UTC m=+594.334334058" Jul 05 19:39:53.402173 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:53.402115 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6hfsl"] Jul 05 19:39:53.404118 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:53.404094 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6hfsl"] Jul 05 19:39:53.470836 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:53.470799 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18df80de-1729-43c8-8ffe-1643f315ab6d" path="/var/lib/kubelet/pods/18df80de-1729-43c8-8ffe-1643f315ab6d/volumes" Jul 05 19:39:53.588091 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:53.588069 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4db5bbd-59d5g" Jul 05 19:39:53.750521 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:53.750487 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4q56z\" (UniqueName: \"kubernetes.io/projected/abff6727-9eea-4d29-88b8-694b70e10e0a-kube-api-access-4q56z\") pod \"abff6727-9eea-4d29-88b8-694b70e10e0a\" (UID: \"abff6727-9eea-4d29-88b8-694b70e10e0a\") " Jul 05 19:39:53.752744 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:53.752716 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abff6727-9eea-4d29-88b8-694b70e10e0a-kube-api-access-4q56z" (OuterVolumeSpecName: "kube-api-access-4q56z") pod "abff6727-9eea-4d29-88b8-694b70e10e0a" (UID: "abff6727-9eea-4d29-88b8-694b70e10e0a"). InnerVolumeSpecName "kube-api-access-4q56z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:39:53.851826 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:53.851792 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q56z\" (UniqueName: \"kubernetes.io/projected/abff6727-9eea-4d29-88b8-694b70e10e0a-kube-api-access-4q56z\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:39:54.325714 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.325682 2573 generic.go:356] "Generic (PLEG): container finished" podID="abff6727-9eea-4d29-88b8-694b70e10e0a" containerID="fd91f232fa5b1e859c5f535689fdab48176e500fe6a913b06b85302d586cfff2" exitCode=0 Jul 05 19:39:54.326187 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.325720 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4db5bbd-59d5g" Jul 05 19:39:54.326187 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.325776 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c4db5bbd-59d5g" event={"ID":"abff6727-9eea-4d29-88b8-694b70e10e0a","Type":"ContainerDied","Data":"fd91f232fa5b1e859c5f535689fdab48176e500fe6a913b06b85302d586cfff2"} Jul 05 19:39:54.326187 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.325818 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c4db5bbd-59d5g" event={"ID":"abff6727-9eea-4d29-88b8-694b70e10e0a","Type":"ContainerDied","Data":"be2966458121eb25d63d6a86a0ff7b2fe02a72f8b3cf66ddb372d03cafc59131"} Jul 05 19:39:54.326187 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.325843 2573 scope.go:117] "RemoveContainer" containerID="fd91f232fa5b1e859c5f535689fdab48176e500fe6a913b06b85302d586cfff2" Jul 05 19:39:54.327444 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.327417 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78b8656c68-zhw78" event={"ID":"e2850f87-838b-48a5-a7fd-d0790268e82e","Type":"ContainerStarted","Data":"dd348a2a42086cd102f33219666a19bcd8f76889d1aae653ba474872c8dd99d4"} Jul 05 19:39:54.334842 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.334827 2573 scope.go:117] "RemoveContainer" containerID="fd91f232fa5b1e859c5f535689fdab48176e500fe6a913b06b85302d586cfff2" Jul 05 19:39:54.335158 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:39:54.335141 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd91f232fa5b1e859c5f535689fdab48176e500fe6a913b06b85302d586cfff2\": container with ID starting with fd91f232fa5b1e859c5f535689fdab48176e500fe6a913b06b85302d586cfff2 not found: ID does not exist" containerID="fd91f232fa5b1e859c5f535689fdab48176e500fe6a913b06b85302d586cfff2" Jul 05 19:39:54.335203 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.335167 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd91f232fa5b1e859c5f535689fdab48176e500fe6a913b06b85302d586cfff2"} err="failed to get container status \"fd91f232fa5b1e859c5f535689fdab48176e500fe6a913b06b85302d586cfff2\": rpc error: code = NotFound desc = could not find container \"fd91f232fa5b1e859c5f535689fdab48176e500fe6a913b06b85302d586cfff2\": container with ID starting with fd91f232fa5b1e859c5f535689fdab48176e500fe6a913b06b85302d586cfff2 not found: ID does not exist" Jul 05 19:39:54.345256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.345216 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-78b8656c68-zhw78" podStartSLOduration=2.022496621 podStartE2EDuration="2.345201197s" totalImagesPullingTime="322.704576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:39:54.343882072 +0000 UTC m=+595.343900187" watchObservedRunningTime="2026-07-05 19:39:54.345201197 +0000 UTC m=+595.345219301" Jul 05 19:39:54.360248 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.360219 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c4db5bbd-59d5g"] Jul 05 19:39:54.361618 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.361599 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c4db5bbd-59d5g"] Jul 05 19:39:54.369819 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.369795 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-76n9m"] Jul 05 19:39:54.369975 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.369956 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-76n9m" podUID="0989624d-8659-40ff-90c3-03ade136a506" containerName="authorino" containerID="cri-o://0288ee4671f27a396a7c17f517556bcade0cb0a0da68ebd33ced20554af26492" gracePeriod=30 Jul 05 19:39:54.622561 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.622533 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-76n9m" Jul 05 19:39:54.760976 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.760942 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nz79\" (UniqueName: \"kubernetes.io/projected/0989624d-8659-40ff-90c3-03ade136a506-kube-api-access-8nz79\") pod \"0989624d-8659-40ff-90c3-03ade136a506\" (UID: \"0989624d-8659-40ff-90c3-03ade136a506\") " Jul 05 19:39:54.763229 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.763205 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0989624d-8659-40ff-90c3-03ade136a506-kube-api-access-8nz79" (OuterVolumeSpecName: "kube-api-access-8nz79") pod "0989624d-8659-40ff-90c3-03ade136a506" (UID: "0989624d-8659-40ff-90c3-03ade136a506"). InnerVolumeSpecName "kube-api-access-8nz79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:39:54.861937 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:54.861849 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nz79\" (UniqueName: \"kubernetes.io/projected/0989624d-8659-40ff-90c3-03ade136a506-kube-api-access-8nz79\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:39:55.333498 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:55.333464 2573 generic.go:356] "Generic (PLEG): container finished" podID="0989624d-8659-40ff-90c3-03ade136a506" containerID="0288ee4671f27a396a7c17f517556bcade0cb0a0da68ebd33ced20554af26492" exitCode=0 Jul 05 19:39:55.333952 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:55.333506 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-76n9m" Jul 05 19:39:55.333952 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:55.333552 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-76n9m" event={"ID":"0989624d-8659-40ff-90c3-03ade136a506","Type":"ContainerDied","Data":"0288ee4671f27a396a7c17f517556bcade0cb0a0da68ebd33ced20554af26492"} Jul 05 19:39:55.333952 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:55.333599 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-76n9m" event={"ID":"0989624d-8659-40ff-90c3-03ade136a506","Type":"ContainerDied","Data":"5374c26c3f3691fca34580ca79aa7d2379e99a1fbcccc263d1376e4bc19106d2"} Jul 05 19:39:55.333952 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:55.333619 2573 scope.go:117] "RemoveContainer" containerID="0288ee4671f27a396a7c17f517556bcade0cb0a0da68ebd33ced20554af26492" Jul 05 19:39:55.342466 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:55.342449 2573 scope.go:117] "RemoveContainer" containerID="0288ee4671f27a396a7c17f517556bcade0cb0a0da68ebd33ced20554af26492" Jul 05 19:39:55.342715 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:39:55.342701 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0288ee4671f27a396a7c17f517556bcade0cb0a0da68ebd33ced20554af26492\": container with ID starting with 0288ee4671f27a396a7c17f517556bcade0cb0a0da68ebd33ced20554af26492 not found: ID does not exist" containerID="0288ee4671f27a396a7c17f517556bcade0cb0a0da68ebd33ced20554af26492" Jul 05 19:39:55.342752 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:55.342722 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0288ee4671f27a396a7c17f517556bcade0cb0a0da68ebd33ced20554af26492"} err="failed to get container status \"0288ee4671f27a396a7c17f517556bcade0cb0a0da68ebd33ced20554af26492\": rpc error: code = NotFound desc = could not find container \"0288ee4671f27a396a7c17f517556bcade0cb0a0da68ebd33ced20554af26492\": container with ID starting with 0288ee4671f27a396a7c17f517556bcade0cb0a0da68ebd33ced20554af26492 not found: ID does not exist" Jul 05 19:39:55.354083 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:55.354027 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-76n9m"] Jul 05 19:39:55.356176 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:55.356156 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-76n9m"] Jul 05 19:39:55.468478 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:55.468429 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0989624d-8659-40ff-90c3-03ade136a506" path="/var/lib/kubelet/pods/0989624d-8659-40ff-90c3-03ade136a506/volumes" Jul 05 19:39:55.468743 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:55.468731 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abff6727-9eea-4d29-88b8-694b70e10e0a" path="/var/lib/kubelet/pods/abff6727-9eea-4d29-88b8-694b70e10e0a/volumes" Jul 05 19:39:58.455262 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.455224 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:58.465558 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.465533 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b9d4db87-zr528"] Jul 05 19:39:58.465905 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.465894 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abff6727-9eea-4d29-88b8-694b70e10e0a" containerName="authorino" Jul 05 19:39:58.465957 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.465907 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="abff6727-9eea-4d29-88b8-694b70e10e0a" containerName="authorino" Jul 05 19:39:58.465957 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.465928 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0989624d-8659-40ff-90c3-03ade136a506" containerName="authorino" Jul 05 19:39:58.465957 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.465933 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0989624d-8659-40ff-90c3-03ade136a506" containerName="authorino" Jul 05 19:39:58.466060 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.465986 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="abff6727-9eea-4d29-88b8-694b70e10e0a" containerName="authorino" Jul 05 19:39:58.466060 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.465995 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0989624d-8659-40ff-90c3-03ade136a506" containerName="authorino" Jul 05 19:39:58.470164 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.470142 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9d4db87-zr528" Jul 05 19:39:58.473256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.473226 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 05 19:39:58.473256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.473252 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 05 19:39:58.473410 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.473263 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-c2849\"" Jul 05 19:39:58.478305 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.478014 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b9d4db87-zr528"] Jul 05 19:39:58.494025 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.493991 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgkcm\" (UniqueName: \"kubernetes.io/projected/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-kube-api-access-zgkcm\") pod \"maas-controller-7b9d4db87-zr528\" (UID: \"2dfae95d-5bde-42cd-a699-f6ec1d0faffc\") " pod="opendatahub/maas-controller-7b9d4db87-zr528" Jul 05 19:39:58.494199 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.494035 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-webhook-cert\") pod \"maas-controller-7b9d4db87-zr528\" (UID: \"2dfae95d-5bde-42cd-a699-f6ec1d0faffc\") " pod="opendatahub/maas-controller-7b9d4db87-zr528" Jul 05 19:39:58.548566 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.548532 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:58.594752 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.594712 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgkcm\" (UniqueName: \"kubernetes.io/projected/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-kube-api-access-zgkcm\") pod \"maas-controller-7b9d4db87-zr528\" (UID: \"2dfae95d-5bde-42cd-a699-f6ec1d0faffc\") " pod="opendatahub/maas-controller-7b9d4db87-zr528" Jul 05 19:39:58.594925 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.594762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-webhook-cert\") pod \"maas-controller-7b9d4db87-zr528\" (UID: \"2dfae95d-5bde-42cd-a699-f6ec1d0faffc\") " pod="opendatahub/maas-controller-7b9d4db87-zr528" Jul 05 19:39:58.594925 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:39:58.594867 2573 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 05 19:39:58.594925 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:39:58.594924 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-webhook-cert podName:2dfae95d-5bde-42cd-a699-f6ec1d0faffc nodeName:}" failed. No retries permitted until 2026-07-05 19:39:59.094905091 +0000 UTC m=+600.094923174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-webhook-cert") pod "maas-controller-7b9d4db87-zr528" (UID: "2dfae95d-5bde-42cd-a699-f6ec1d0faffc") : secret "maas-controller-webhook-cert" not found Jul 05 19:39:58.605819 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.605791 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgkcm\" (UniqueName: \"kubernetes.io/projected/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-kube-api-access-zgkcm\") pod \"maas-controller-7b9d4db87-zr528\" (UID: \"2dfae95d-5bde-42cd-a699-f6ec1d0faffc\") " pod="opendatahub/maas-controller-7b9d4db87-zr528" Jul 05 19:39:58.626570 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.626543 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:58.630329 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.630305 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f64fbc48c-t6plz"] Jul 05 19:39:58.633695 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.633680 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f64fbc48c-t6plz" Jul 05 19:39:58.642488 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.642462 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f64fbc48c-t6plz"] Jul 05 19:39:58.695235 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.695205 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4qm\" (UniqueName: \"kubernetes.io/projected/f8a3885c-9f3f-43dd-b156-046739fc7e16-kube-api-access-fz4qm\") pod \"maas-controller-6f64fbc48c-t6plz\" (UID: \"f8a3885c-9f3f-43dd-b156-046739fc7e16\") " pod="opendatahub/maas-controller-6f64fbc48c-t6plz" Jul 05 19:39:58.695409 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.695253 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8a3885c-9f3f-43dd-b156-046739fc7e16-webhook-cert\") pod \"maas-controller-6f64fbc48c-t6plz\" (UID: \"f8a3885c-9f3f-43dd-b156-046739fc7e16\") " pod="opendatahub/maas-controller-6f64fbc48c-t6plz" Jul 05 19:39:58.796711 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.796660 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4qm\" (UniqueName: \"kubernetes.io/projected/f8a3885c-9f3f-43dd-b156-046739fc7e16-kube-api-access-fz4qm\") pod \"maas-controller-6f64fbc48c-t6plz\" (UID: \"f8a3885c-9f3f-43dd-b156-046739fc7e16\") " pod="opendatahub/maas-controller-6f64fbc48c-t6plz" Jul 05 19:39:58.796711 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.796727 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8a3885c-9f3f-43dd-b156-046739fc7e16-webhook-cert\") pod \"maas-controller-6f64fbc48c-t6plz\" (UID: \"f8a3885c-9f3f-43dd-b156-046739fc7e16\") " pod="opendatahub/maas-controller-6f64fbc48c-t6plz" Jul 05 19:39:58.796995 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:39:58.796817 2573 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 05 19:39:58.796995 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:39:58.796883 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a3885c-9f3f-43dd-b156-046739fc7e16-webhook-cert podName:f8a3885c-9f3f-43dd-b156-046739fc7e16 nodeName:}" failed. No retries permitted until 2026-07-05 19:39:59.296868635 +0000 UTC m=+600.296886718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/f8a3885c-9f3f-43dd-b156-046739fc7e16-webhook-cert") pod "maas-controller-6f64fbc48c-t6plz" (UID: "f8a3885c-9f3f-43dd-b156-046739fc7e16") : secret "maas-controller-webhook-cert" not found Jul 05 19:39:58.807929 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:58.807906 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4qm\" (UniqueName: \"kubernetes.io/projected/f8a3885c-9f3f-43dd-b156-046739fc7e16-kube-api-access-fz4qm\") pod \"maas-controller-6f64fbc48c-t6plz\" (UID: \"f8a3885c-9f3f-43dd-b156-046739fc7e16\") " pod="opendatahub/maas-controller-6f64fbc48c-t6plz" Jul 05 19:39:59.099250 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:59.099157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-webhook-cert\") pod \"maas-controller-7b9d4db87-zr528\" (UID: \"2dfae95d-5bde-42cd-a699-f6ec1d0faffc\") " pod="opendatahub/maas-controller-7b9d4db87-zr528" Jul 05 19:39:59.099420 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:39:59.099320 2573 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 05 19:39:59.099420 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:39:59.099391 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-webhook-cert podName:2dfae95d-5bde-42cd-a699-f6ec1d0faffc nodeName:}" failed. No retries permitted until 2026-07-05 19:40:00.099370516 +0000 UTC m=+601.099388601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-webhook-cert") pod "maas-controller-7b9d4db87-zr528" (UID: "2dfae95d-5bde-42cd-a699-f6ec1d0faffc") : secret "maas-controller-webhook-cert" not found Jul 05 19:39:59.301305 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:59.301266 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8a3885c-9f3f-43dd-b156-046739fc7e16-webhook-cert\") pod \"maas-controller-6f64fbc48c-t6plz\" (UID: \"f8a3885c-9f3f-43dd-b156-046739fc7e16\") " pod="opendatahub/maas-controller-6f64fbc48c-t6plz" Jul 05 19:39:59.301535 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:39:59.301436 2573 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 05 19:39:59.301535 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:39:59.301507 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a3885c-9f3f-43dd-b156-046739fc7e16-webhook-cert podName:f8a3885c-9f3f-43dd-b156-046739fc7e16 nodeName:}" failed. No retries permitted until 2026-07-05 19:40:00.301488371 +0000 UTC m=+601.301506454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/f8a3885c-9f3f-43dd-b156-046739fc7e16-webhook-cert") pod "maas-controller-6f64fbc48c-t6plz" (UID: "f8a3885c-9f3f-43dd-b156-046739fc7e16") : secret "maas-controller-webhook-cert" not found Jul 05 19:39:59.340740 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:59.340701 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:39:59.448910 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:59.448827 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 19:39:59.449314 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:39:59.449296 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 19:40:00.108936 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:00.108898 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-webhook-cert\") pod \"maas-controller-7b9d4db87-zr528\" (UID: \"2dfae95d-5bde-42cd-a699-f6ec1d0faffc\") " pod="opendatahub/maas-controller-7b9d4db87-zr528" Jul 05 19:40:00.111455 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:00.111433 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-webhook-cert\") pod \"maas-controller-7b9d4db87-zr528\" (UID: \"2dfae95d-5bde-42cd-a699-f6ec1d0faffc\") " pod="opendatahub/maas-controller-7b9d4db87-zr528" Jul 05 19:40:00.284908 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:00.284871 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-c2849\"" Jul 05 19:40:00.292190 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:00.292169 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9d4db87-zr528" Jul 05 19:40:00.310729 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:00.310702 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8a3885c-9f3f-43dd-b156-046739fc7e16-webhook-cert\") pod \"maas-controller-6f64fbc48c-t6plz\" (UID: \"f8a3885c-9f3f-43dd-b156-046739fc7e16\") " pod="opendatahub/maas-controller-6f64fbc48c-t6plz" Jul 05 19:40:00.313202 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:00.313181 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8a3885c-9f3f-43dd-b156-046739fc7e16-webhook-cert\") pod \"maas-controller-6f64fbc48c-t6plz\" (UID: \"f8a3885c-9f3f-43dd-b156-046739fc7e16\") " pod="opendatahub/maas-controller-6f64fbc48c-t6plz" Jul 05 19:40:00.424189 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:00.424164 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b9d4db87-zr528"] Jul 05 19:40:00.426862 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:40:00.426836 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfae95d_5bde_42cd_a699_f6ec1d0faffc.slice/crio-f1268e97fa330575686bbbc44be5c24a90402ddf872b35c56c749b78b6cafeb2 WatchSource:0}: Error finding container f1268e97fa330575686bbbc44be5c24a90402ddf872b35c56c749b78b6cafeb2: Status 404 returned error can't find the container with id f1268e97fa330575686bbbc44be5c24a90402ddf872b35c56c749b78b6cafeb2 Jul 05 19:40:00.445421 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:00.445391 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f64fbc48c-t6plz" Jul 05 19:40:00.579943 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:00.579918 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f64fbc48c-t6plz"] Jul 05 19:40:00.581670 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:40:00.581643 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a3885c_9f3f_43dd_b156_046739fc7e16.slice/crio-0f097fe5064fe885b5af7dcda8955654a44522c326de7cc606aa59704794c22d WatchSource:0}: Error finding container 0f097fe5064fe885b5af7dcda8955654a44522c326de7cc606aa59704794c22d: Status 404 returned error can't find the container with id 0f097fe5064fe885b5af7dcda8955654a44522c326de7cc606aa59704794c22d Jul 05 19:40:01.370478 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:01.370429 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9d4db87-zr528" event={"ID":"2dfae95d-5bde-42cd-a699-f6ec1d0faffc","Type":"ContainerStarted","Data":"f1268e97fa330575686bbbc44be5c24a90402ddf872b35c56c749b78b6cafeb2"} Jul 05 19:40:01.373054 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:01.372748 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f64fbc48c-t6plz" event={"ID":"f8a3885c-9f3f-43dd-b156-046739fc7e16","Type":"ContainerStarted","Data":"0f097fe5064fe885b5af7dcda8955654a44522c326de7cc606aa59704794c22d"} Jul 05 19:40:03.254391 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:03.254367 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 05 19:40:03.385111 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:03.385076 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9d4db87-zr528" event={"ID":"2dfae95d-5bde-42cd-a699-f6ec1d0faffc","Type":"ContainerStarted","Data":"d18be778cfc54e513f8da61ea8a28795b604670535726ae1c56f317884df3b44"} Jul 05 19:40:03.385325 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:03.385238 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b9d4db87-zr528" Jul 05 19:40:03.386512 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:03.386486 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f64fbc48c-t6plz" event={"ID":"f8a3885c-9f3f-43dd-b156-046739fc7e16","Type":"ContainerStarted","Data":"75caf57a3ad63d8738829055f2fa6845c53ed3515fd5d9093b54a864be27ee77"} Jul 05 19:40:03.386624 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:03.386570 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f64fbc48c-t6plz" Jul 05 19:40:03.401958 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:03.401911 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b9d4db87-zr528" podStartSLOduration=2.579043327 podStartE2EDuration="5.401896389s" totalImagesPullingTime="2.822853062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:40:03.400468275 +0000 UTC m=+604.400486380" watchObservedRunningTime="2026-07-05 19:40:03.401896389 +0000 UTC m=+604.401914494" Jul 05 19:40:03.421203 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:03.421143 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f64fbc48c-t6plz" podStartSLOduration=2.745345721 podStartE2EDuration="5.421127268s" totalImagesPullingTime="2.675781547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:40:03.4202032 +0000 UTC m=+604.420221318" watchObservedRunningTime="2026-07-05 19:40:03.421127268 +0000 UTC m=+604.421145373" Jul 05 19:40:03.507526 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:03.507493 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:40:05.195564 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:05.195517 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:40:14.395977 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:14.395950 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b9d4db87-zr528" Jul 05 19:40:14.396442 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:14.396015 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f64fbc48c-t6plz" Jul 05 19:40:14.441943 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:14.441905 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b9d4db87-zr528"] Jul 05 19:40:14.442611 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:14.442554 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7b9d4db87-zr528" podUID="2dfae95d-5bde-42cd-a699-f6ec1d0faffc" containerName="manager" containerID="cri-o://d18be778cfc54e513f8da61ea8a28795b604670535726ae1c56f317884df3b44" gracePeriod=30 Jul 05 19:40:14.692014 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:14.691987 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9d4db87-zr528" Jul 05 19:40:14.747717 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:14.747682 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-webhook-cert\") pod \"2dfae95d-5bde-42cd-a699-f6ec1d0faffc\" (UID: \"2dfae95d-5bde-42cd-a699-f6ec1d0faffc\") " Jul 05 19:40:14.747906 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:14.747739 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgkcm\" (UniqueName: \"kubernetes.io/projected/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-kube-api-access-zgkcm\") pod \"2dfae95d-5bde-42cd-a699-f6ec1d0faffc\" (UID: \"2dfae95d-5bde-42cd-a699-f6ec1d0faffc\") " Jul 05 19:40:14.750023 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:14.749996 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2dfae95d-5bde-42cd-a699-f6ec1d0faffc" (UID: "2dfae95d-5bde-42cd-a699-f6ec1d0faffc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:40:14.750155 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:14.750080 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-kube-api-access-zgkcm" (OuterVolumeSpecName: "kube-api-access-zgkcm") pod "2dfae95d-5bde-42cd-a699-f6ec1d0faffc" (UID: "2dfae95d-5bde-42cd-a699-f6ec1d0faffc"). InnerVolumeSpecName "kube-api-access-zgkcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:40:14.848671 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:14.848636 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgkcm\" (UniqueName: \"kubernetes.io/projected/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-kube-api-access-zgkcm\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:40:14.848671 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:14.848667 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2dfae95d-5bde-42cd-a699-f6ec1d0faffc-webhook-cert\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:40:15.448612 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:15.448574 2573 generic.go:356] "Generic (PLEG): container finished" podID="2dfae95d-5bde-42cd-a699-f6ec1d0faffc" containerID="d18be778cfc54e513f8da61ea8a28795b604670535726ae1c56f317884df3b44" exitCode=0 Jul 05 19:40:15.449007 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:15.448619 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b9d4db87-zr528" Jul 05 19:40:15.449007 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:15.448668 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9d4db87-zr528" event={"ID":"2dfae95d-5bde-42cd-a699-f6ec1d0faffc","Type":"ContainerDied","Data":"d18be778cfc54e513f8da61ea8a28795b604670535726ae1c56f317884df3b44"} Jul 05 19:40:15.449007 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:15.448714 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b9d4db87-zr528" event={"ID":"2dfae95d-5bde-42cd-a699-f6ec1d0faffc","Type":"ContainerDied","Data":"f1268e97fa330575686bbbc44be5c24a90402ddf872b35c56c749b78b6cafeb2"} Jul 05 19:40:15.449007 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:15.448734 2573 scope.go:117] "RemoveContainer" containerID="d18be778cfc54e513f8da61ea8a28795b604670535726ae1c56f317884df3b44" Jul 05 19:40:15.457713 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:15.457692 2573 scope.go:117] "RemoveContainer" containerID="d18be778cfc54e513f8da61ea8a28795b604670535726ae1c56f317884df3b44" Jul 05 19:40:15.457971 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:40:15.457953 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18be778cfc54e513f8da61ea8a28795b604670535726ae1c56f317884df3b44\": container with ID starting with d18be778cfc54e513f8da61ea8a28795b604670535726ae1c56f317884df3b44 not found: ID does not exist" containerID="d18be778cfc54e513f8da61ea8a28795b604670535726ae1c56f317884df3b44" Jul 05 19:40:15.458009 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:15.457980 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18be778cfc54e513f8da61ea8a28795b604670535726ae1c56f317884df3b44"} err="failed to get container status \"d18be778cfc54e513f8da61ea8a28795b604670535726ae1c56f317884df3b44\": rpc error: code = NotFound desc = could not find container \"d18be778cfc54e513f8da61ea8a28795b604670535726ae1c56f317884df3b44\": container with ID starting with d18be778cfc54e513f8da61ea8a28795b604670535726ae1c56f317884df3b44 not found: ID does not exist" Jul 05 19:40:15.469458 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:15.469389 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b9d4db87-zr528"] Jul 05 19:40:15.470891 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:15.470867 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b9d4db87-zr528"] Jul 05 19:40:17.470408 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:17.470378 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dfae95d-5bde-42cd-a699-f6ec1d0faffc" path="/var/lib/kubelet/pods/2dfae95d-5bde-42cd-a699-f6ec1d0faffc/volumes" Jul 05 19:40:35.984340 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:35.984259 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:40:36.430570 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.430537 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:40:36.475575 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.475541 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-6d694dc8c4-4kgb2"] Jul 05 19:40:36.475921 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.475909 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dfae95d-5bde-42cd-a699-f6ec1d0faffc" containerName="manager" Jul 05 19:40:36.476007 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.475923 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dfae95d-5bde-42cd-a699-f6ec1d0faffc" containerName="manager" Jul 05 19:40:36.476007 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.475979 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dfae95d-5bde-42cd-a699-f6ec1d0faffc" containerName="manager" Jul 05 19:40:36.480663 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.480644 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-6d694dc8c4-4kgb2" Jul 05 19:40:36.483711 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.483683 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 05 19:40:36.483711 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.483702 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-7xc7w\"" Jul 05 19:40:36.490573 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.490551 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-6d694dc8c4-4kgb2"] Jul 05 19:40:36.549410 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.549381 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c10c543-1810-4d6d-8b1d-e1afd66c88eb-maas-api-tls\") pod \"maas-api-6d694dc8c4-4kgb2\" (UID: \"1c10c543-1810-4d6d-8b1d-e1afd66c88eb\") " pod="opendatahub/maas-api-6d694dc8c4-4kgb2" Jul 05 19:40:36.549586 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.549426 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c10c543-1810-4d6d-8b1d-e1afd66c88eb-openshift-service-ca\") pod \"maas-api-6d694dc8c4-4kgb2\" (UID: \"1c10c543-1810-4d6d-8b1d-e1afd66c88eb\") " pod="opendatahub/maas-api-6d694dc8c4-4kgb2" Jul 05 19:40:36.549586 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.549497 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gr66\" (UniqueName: \"kubernetes.io/projected/1c10c543-1810-4d6d-8b1d-e1afd66c88eb-kube-api-access-6gr66\") pod \"maas-api-6d694dc8c4-4kgb2\" (UID: \"1c10c543-1810-4d6d-8b1d-e1afd66c88eb\") " pod="opendatahub/maas-api-6d694dc8c4-4kgb2" Jul 05 19:40:36.650754 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.650721 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gr66\" (UniqueName: \"kubernetes.io/projected/1c10c543-1810-4d6d-8b1d-e1afd66c88eb-kube-api-access-6gr66\") pod \"maas-api-6d694dc8c4-4kgb2\" (UID: \"1c10c543-1810-4d6d-8b1d-e1afd66c88eb\") " pod="opendatahub/maas-api-6d694dc8c4-4kgb2" Jul 05 19:40:36.650966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.650795 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c10c543-1810-4d6d-8b1d-e1afd66c88eb-maas-api-tls\") pod \"maas-api-6d694dc8c4-4kgb2\" (UID: \"1c10c543-1810-4d6d-8b1d-e1afd66c88eb\") " pod="opendatahub/maas-api-6d694dc8c4-4kgb2" Jul 05 19:40:36.650966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.650825 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c10c543-1810-4d6d-8b1d-e1afd66c88eb-openshift-service-ca\") pod \"maas-api-6d694dc8c4-4kgb2\" (UID: \"1c10c543-1810-4d6d-8b1d-e1afd66c88eb\") " pod="opendatahub/maas-api-6d694dc8c4-4kgb2" Jul 05 19:40:36.650966 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:40:36.650933 2573 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 05 19:40:36.651177 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:40:36.650993 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c10c543-1810-4d6d-8b1d-e1afd66c88eb-maas-api-tls podName:1c10c543-1810-4d6d-8b1d-e1afd66c88eb nodeName:}" failed. No retries permitted until 2026-07-05 19:40:37.150976826 +0000 UTC m=+638.150994909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1c10c543-1810-4d6d-8b1d-e1afd66c88eb-maas-api-tls") pod "maas-api-6d694dc8c4-4kgb2" (UID: "1c10c543-1810-4d6d-8b1d-e1afd66c88eb") : secret "maas-api-serving-cert" not found Jul 05 19:40:36.651474 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.651448 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c10c543-1810-4d6d-8b1d-e1afd66c88eb-openshift-service-ca\") pod \"maas-api-6d694dc8c4-4kgb2\" (UID: \"1c10c543-1810-4d6d-8b1d-e1afd66c88eb\") " pod="opendatahub/maas-api-6d694dc8c4-4kgb2" Jul 05 19:40:36.662594 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.662563 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gr66\" (UniqueName: \"kubernetes.io/projected/1c10c543-1810-4d6d-8b1d-e1afd66c88eb-kube-api-access-6gr66\") pod \"maas-api-6d694dc8c4-4kgb2\" (UID: \"1c10c543-1810-4d6d-8b1d-e1afd66c88eb\") " pod="opendatahub/maas-api-6d694dc8c4-4kgb2" Jul 05 19:40:36.708198 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.708112 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:40:36.850941 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.850898 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:40:36.942795 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.942763 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:40:36.943920 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.943899 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-rkn5g"] Jul 05 19:40:36.946868 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.946852 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-rkn5g" Jul 05 19:40:36.949454 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.949434 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 05 19:40:36.949560 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.949539 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-njz2x\"" Jul 05 19:40:36.953498 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.953477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0258ded4-4884-4e4c-9124-8b9e227b222f-plugins-config-volume\") pod \"payload-pre-processing-7964457696-rkn5g\" (UID: \"0258ded4-4884-4e4c-9124-8b9e227b222f\") " pod="openshift-ingress/payload-pre-processing-7964457696-rkn5g" Jul 05 19:40:36.953633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.953613 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr7hh\" (UniqueName: \"kubernetes.io/projected/0258ded4-4884-4e4c-9124-8b9e227b222f-kube-api-access-tr7hh\") pod \"payload-pre-processing-7964457696-rkn5g\" (UID: \"0258ded4-4884-4e4c-9124-8b9e227b222f\") " pod="openshift-ingress/payload-pre-processing-7964457696-rkn5g" Jul 05 19:40:36.957859 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:36.957839 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-rkn5g"] Jul 05 19:40:37.054960 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:37.054926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr7hh\" (UniqueName: \"kubernetes.io/projected/0258ded4-4884-4e4c-9124-8b9e227b222f-kube-api-access-tr7hh\") pod \"payload-pre-processing-7964457696-rkn5g\" (UID: \"0258ded4-4884-4e4c-9124-8b9e227b222f\") " pod="openshift-ingress/payload-pre-processing-7964457696-rkn5g" Jul 05 19:40:37.055352 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:37.054983 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0258ded4-4884-4e4c-9124-8b9e227b222f-plugins-config-volume\") pod \"payload-pre-processing-7964457696-rkn5g\" (UID: \"0258ded4-4884-4e4c-9124-8b9e227b222f\") " pod="openshift-ingress/payload-pre-processing-7964457696-rkn5g" Jul 05 19:40:37.055707 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:37.055689 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0258ded4-4884-4e4c-9124-8b9e227b222f-plugins-config-volume\") pod \"payload-pre-processing-7964457696-rkn5g\" (UID: \"0258ded4-4884-4e4c-9124-8b9e227b222f\") " pod="openshift-ingress/payload-pre-processing-7964457696-rkn5g" Jul 05 19:40:37.065333 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:37.065306 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr7hh\" (UniqueName: \"kubernetes.io/projected/0258ded4-4884-4e4c-9124-8b9e227b222f-kube-api-access-tr7hh\") pod \"payload-pre-processing-7964457696-rkn5g\" (UID: \"0258ded4-4884-4e4c-9124-8b9e227b222f\") " pod="openshift-ingress/payload-pre-processing-7964457696-rkn5g" Jul 05 19:40:37.156388 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:37.156346 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c10c543-1810-4d6d-8b1d-e1afd66c88eb-maas-api-tls\") pod \"maas-api-6d694dc8c4-4kgb2\" (UID: \"1c10c543-1810-4d6d-8b1d-e1afd66c88eb\") " pod="opendatahub/maas-api-6d694dc8c4-4kgb2" Jul 05 19:40:37.158818 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:37.158788 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c10c543-1810-4d6d-8b1d-e1afd66c88eb-maas-api-tls\") pod \"maas-api-6d694dc8c4-4kgb2\" (UID: \"1c10c543-1810-4d6d-8b1d-e1afd66c88eb\") " pod="opendatahub/maas-api-6d694dc8c4-4kgb2" Jul 05 19:40:37.258065 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:37.257998 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-rkn5g" Jul 05 19:40:37.391325 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:37.391294 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-6d694dc8c4-4kgb2" Jul 05 19:40:37.392397 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:37.392373 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-rkn5g"] Jul 05 19:40:37.394885 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:40:37.394860 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0258ded4_4884_4e4c_9124_8b9e227b222f.slice/crio-2bef2867df2580ba8e7970bddafe848906d54d244fc7385407a4ac131561487e WatchSource:0}: Error finding container 2bef2867df2580ba8e7970bddafe848906d54d244fc7385407a4ac131561487e: Status 404 returned error can't find the container with id 2bef2867df2580ba8e7970bddafe848906d54d244fc7385407a4ac131561487e Jul 05 19:40:37.525585 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:37.525559 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-6d694dc8c4-4kgb2"] Jul 05 19:40:37.527548 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:40:37.527517 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c10c543_1810_4d6d_8b1d_e1afd66c88eb.slice/crio-afa66a87722e2811549916e93694857a55ef7e34d7ed27af2a618e54f1637706 WatchSource:0}: Error finding container afa66a87722e2811549916e93694857a55ef7e34d7ed27af2a618e54f1637706: Status 404 returned error can't find the container with id afa66a87722e2811549916e93694857a55ef7e34d7ed27af2a618e54f1637706 Jul 05 19:40:37.556537 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:37.556494 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-6d694dc8c4-4kgb2" event={"ID":"1c10c543-1810-4d6d-8b1d-e1afd66c88eb","Type":"ContainerStarted","Data":"afa66a87722e2811549916e93694857a55ef7e34d7ed27af2a618e54f1637706"} Jul 05 19:40:37.557585 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:37.557562 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-rkn5g" event={"ID":"0258ded4-4884-4e4c-9124-8b9e227b222f","Type":"ContainerStarted","Data":"2bef2867df2580ba8e7970bddafe848906d54d244fc7385407a4ac131561487e"} Jul 05 19:40:39.568520 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:39.568419 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-rkn5g" event={"ID":"0258ded4-4884-4e4c-9124-8b9e227b222f","Type":"ContainerStarted","Data":"cdcd7b39f25f6dcec8e908cf9290308ffb67f5978653e16b48cca67a3afdabb0"} Jul 05 19:40:39.568520 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:39.568495 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-rkn5g" Jul 05 19:40:39.569814 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:39.569791 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-6d694dc8c4-4kgb2" event={"ID":"1c10c543-1810-4d6d-8b1d-e1afd66c88eb","Type":"ContainerStarted","Data":"40deb0f90ea61142a107d9527cd61e908e2f77ef6626474887575a43e18dde0c"} Jul 05 19:40:39.569922 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:39.569908 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-6d694dc8c4-4kgb2" Jul 05 19:40:39.585072 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:39.585011 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-rkn5g" podStartSLOduration=2.310714544 podStartE2EDuration="3.584997891s" totalImagesPullingTime="1.274283347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:40:39.58277463 +0000 UTC m=+640.582792735" watchObservedRunningTime="2026-07-05 19:40:39.584997891 +0000 UTC m=+640.585016004" Jul 05 19:40:39.599001 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:39.598952 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-6d694dc8c4-4kgb2" podStartSLOduration=1.863653372 podStartE2EDuration="3.598937365s" totalImagesPullingTime="1.735283993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:40:39.597370515 +0000 UTC m=+640.597388620" watchObservedRunningTime="2026-07-05 19:40:39.598937365 +0000 UTC m=+640.598955469" Jul 05 19:40:45.579480 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:45.579445 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-6d694dc8c4-4kgb2" Jul 05 19:40:46.537760 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:46.537720 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:40:46.634517 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:46.634481 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:40:50.577195 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:50.577152 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-rkn5g" Jul 05 19:40:51.257733 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.257692 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd"] Jul 05 19:40:51.261535 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.261516 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.264265 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.264244 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 05 19:40:51.264365 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.264296 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cxv76\"" Jul 05 19:40:51.265493 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.265478 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 05 19:40:51.265541 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.265484 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 05 19:40:51.270002 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.269980 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd"] Jul 05 19:40:51.298583 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.298549 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:40:51.399068 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.398997 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e03934f-dded-4ad0-99e7-92cc0f843a1d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.399249 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.399167 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e03934f-dded-4ad0-99e7-92cc0f843a1d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.399249 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.399200 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e03934f-dded-4ad0-99e7-92cc0f843a1d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.399249 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.399227 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e03934f-dded-4ad0-99e7-92cc0f843a1d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.399352 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.399265 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e03934f-dded-4ad0-99e7-92cc0f843a1d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.399352 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.399290 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lngpf\" (UniqueName: \"kubernetes.io/projected/2e03934f-dded-4ad0-99e7-92cc0f843a1d-kube-api-access-lngpf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.499981 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.499944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e03934f-dded-4ad0-99e7-92cc0f843a1d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.499981 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.499980 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e03934f-dded-4ad0-99e7-92cc0f843a1d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.500231 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.500002 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e03934f-dded-4ad0-99e7-92cc0f843a1d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.500231 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.500155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e03934f-dded-4ad0-99e7-92cc0f843a1d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.500231 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.500208 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lngpf\" (UniqueName: \"kubernetes.io/projected/2e03934f-dded-4ad0-99e7-92cc0f843a1d-kube-api-access-lngpf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.500368 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.500276 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e03934f-dded-4ad0-99e7-92cc0f843a1d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.500417 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.500381 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e03934f-dded-4ad0-99e7-92cc0f843a1d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.500469 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.500422 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e03934f-dded-4ad0-99e7-92cc0f843a1d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.500661 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.500642 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e03934f-dded-4ad0-99e7-92cc0f843a1d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.502347 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.502327 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e03934f-dded-4ad0-99e7-92cc0f843a1d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.502630 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.502613 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e03934f-dded-4ad0-99e7-92cc0f843a1d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.509221 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.509165 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lngpf\" (UniqueName: \"kubernetes.io/projected/2e03934f-dded-4ad0-99e7-92cc0f843a1d-kube-api-access-lngpf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd\" (UID: \"2e03934f-dded-4ad0-99e7-92cc0f843a1d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.572465 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.572416 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:40:51.706721 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:51.706693 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd"] Jul 05 19:40:51.709926 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:40:51.709897 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e03934f_dded_4ad0_99e7_92cc0f843a1d.slice/crio-ab2d03199b9b4ad4ff5c65f841ebcae68be30e24c94672fbd388369bf1582e2e WatchSource:0}: Error finding container ab2d03199b9b4ad4ff5c65f841ebcae68be30e24c94672fbd388369bf1582e2e: Status 404 returned error can't find the container with id ab2d03199b9b4ad4ff5c65f841ebcae68be30e24c94672fbd388369bf1582e2e Jul 05 19:40:52.640842 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:52.640800 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" event={"ID":"2e03934f-dded-4ad0-99e7-92cc0f843a1d","Type":"ContainerStarted","Data":"ab2d03199b9b4ad4ff5c65f841ebcae68be30e24c94672fbd388369bf1582e2e"} Jul 05 19:40:53.647060 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:53.647000 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" event={"ID":"2e03934f-dded-4ad0-99e7-92cc0f843a1d","Type":"ContainerStarted","Data":"b64cc7278b3195434e1694fd3d505763c11f653257cd6962c1b4daa85d166f85"} Jul 05 19:40:53.664480 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:53.664429 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" podStartSLOduration=1.5675313119999998 podStartE2EDuration="2.664415314s" totalImagesPullingTime="1.096884002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:40:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:40:53.663332008 +0000 UTC m=+654.663350116" watchObservedRunningTime="2026-07-05 19:40:53.664415314 +0000 UTC m=+654.664433418" Jul 05 19:40:59.162607 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:40:59.162559 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:41:01.572832 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:01.572797 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:41:01.573234 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:01.572864 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:41:01.585169 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:01.585147 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:41:01.695243 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:01.695213 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd" Jul 05 19:41:13.547207 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.547176 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-92skn"] Jul 05 19:41:13.552782 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.552763 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.555488 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.555469 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 05 19:41:13.560586 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.560566 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-92skn"] Jul 05 19:41:13.606683 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.606651 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15d38acb-947e-4c2d-b751-df98c72781b6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.606683 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.606691 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15d38acb-947e-4c2d-b751-df98c72781b6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.606917 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.606778 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15d38acb-947e-4c2d-b751-df98c72781b6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.606917 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.606805 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15d38acb-947e-4c2d-b751-df98c72781b6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.606917 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.606881 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15d38acb-947e-4c2d-b751-df98c72781b6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.606917 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.606900 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnkkl\" (UniqueName: \"kubernetes.io/projected/15d38acb-947e-4c2d-b751-df98c72781b6-kube-api-access-wnkkl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.652651 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.652607 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:41:13.707468 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.707424 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15d38acb-947e-4c2d-b751-df98c72781b6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.707642 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.707477 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15d38acb-947e-4c2d-b751-df98c72781b6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.707642 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.707513 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15d38acb-947e-4c2d-b751-df98c72781b6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.707642 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.707535 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnkkl\" (UniqueName: \"kubernetes.io/projected/15d38acb-947e-4c2d-b751-df98c72781b6-kube-api-access-wnkkl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.707642 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.707597 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15d38acb-947e-4c2d-b751-df98c72781b6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.707642 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.707613 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15d38acb-947e-4c2d-b751-df98c72781b6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.707896 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.707863 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15d38acb-947e-4c2d-b751-df98c72781b6-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.708028 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.708002 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15d38acb-947e-4c2d-b751-df98c72781b6-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.708139 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.708117 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15d38acb-947e-4c2d-b751-df98c72781b6-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.710072 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.710053 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15d38acb-947e-4c2d-b751-df98c72781b6-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.710248 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.710232 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15d38acb-947e-4c2d-b751-df98c72781b6-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.715486 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.715470 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnkkl\" (UniqueName: \"kubernetes.io/projected/15d38acb-947e-4c2d-b751-df98c72781b6-kube-api-access-wnkkl\") pod \"e2e-trlp-test-simulated-kserve-598466548d-92skn\" (UID: \"15d38acb-947e-4c2d-b751-df98c72781b6\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:13.863896 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:13.863814 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:14.002769 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:14.002742 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-92skn"] Jul 05 19:41:14.004740 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:41:14.004712 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d38acb_947e_4c2d_b751_df98c72781b6.slice/crio-d248aafe8ae93de2b14c9e113fd63b34e33f8565b2331b4ee3918022cdb0f02c WatchSource:0}: Error finding container d248aafe8ae93de2b14c9e113fd63b34e33f8565b2331b4ee3918022cdb0f02c: Status 404 returned error can't find the container with id d248aafe8ae93de2b14c9e113fd63b34e33f8565b2331b4ee3918022cdb0f02c Jul 05 19:41:14.751572 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:14.751532 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" event={"ID":"15d38acb-947e-4c2d-b751-df98c72781b6","Type":"ContainerStarted","Data":"7c4b319a77936411fcc9cb247ef52700f3766ed11426027ef0df856a61e9e093"} Jul 05 19:41:14.752003 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:14.751622 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" event={"ID":"15d38acb-947e-4c2d-b751-df98c72781b6","Type":"ContainerStarted","Data":"d248aafe8ae93de2b14c9e113fd63b34e33f8565b2331b4ee3918022cdb0f02c"} Jul 05 19:41:14.767647 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:14.767592 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" podStartSLOduration=1.598445061 podStartE2EDuration="1.767574938s" totalImagesPullingTime="169.129877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:41:14.767071236 +0000 UTC m=+675.767089339" watchObservedRunningTime="2026-07-05 19:41:14.767574938 +0000 UTC m=+675.767593041" Jul 05 19:41:18.668378 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:18.668332 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:41:23.864008 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:23.863969 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:23.864475 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:23.864071 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:23.875881 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:23.875856 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:24.809667 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:24.809635 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-92skn" Jul 05 19:41:28.939386 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:28.939346 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:41:54.072358 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:54.072278 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:41:58.628486 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:58.628453 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:41:58.728646 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:41:58.728611 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:42:54.154342 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:42:54.154305 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:42:59.907855 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:42:59.907816 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b5b7fd64f-vsx9f"] Jul 05 19:42:59.911512 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:42:59.911489 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b5b7fd64f-vsx9f" Jul 05 19:42:59.919303 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:42:59.919235 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b5b7fd64f-vsx9f"] Jul 05 19:42:59.996192 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:42:59.996145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2-tls-cert\") pod \"authorino-6b5b7fd64f-vsx9f\" (UID: \"0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2\") " pod="kuadrant-system/authorino-6b5b7fd64f-vsx9f" Jul 05 19:42:59.996370 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:42:59.996289 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q675\" (UniqueName: \"kubernetes.io/projected/0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2-kube-api-access-5q675\") pod \"authorino-6b5b7fd64f-vsx9f\" (UID: \"0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2\") " pod="kuadrant-system/authorino-6b5b7fd64f-vsx9f" Jul 05 19:43:00.096989 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:00.096944 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q675\" (UniqueName: \"kubernetes.io/projected/0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2-kube-api-access-5q675\") pod \"authorino-6b5b7fd64f-vsx9f\" (UID: \"0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2\") " pod="kuadrant-system/authorino-6b5b7fd64f-vsx9f" Jul 05 19:43:00.097203 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:00.097019 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2-tls-cert\") pod \"authorino-6b5b7fd64f-vsx9f\" (UID: \"0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2\") " pod="kuadrant-system/authorino-6b5b7fd64f-vsx9f" Jul 05 19:43:00.099732 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:00.099706 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2-tls-cert\") pod \"authorino-6b5b7fd64f-vsx9f\" (UID: \"0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2\") " pod="kuadrant-system/authorino-6b5b7fd64f-vsx9f" Jul 05 19:43:00.104718 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:00.104691 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q675\" (UniqueName: \"kubernetes.io/projected/0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2-kube-api-access-5q675\") pod \"authorino-6b5b7fd64f-vsx9f\" (UID: \"0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2\") " pod="kuadrant-system/authorino-6b5b7fd64f-vsx9f" Jul 05 19:43:00.221985 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:00.221890 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b5b7fd64f-vsx9f" Jul 05 19:43:00.300838 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:00.300748 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:43:00.358098 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:00.358037 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b5b7fd64f-vsx9f"] Jul 05 19:43:00.360345 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:43:00.360320 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0da6f7e0_c5cb_402d_b4c4_c42cb443ecf2.slice/crio-3f58a0dc2bf54e80a364bae0685d8f8e7bf4412727f6ed72b63ba6e0137ca0f4 WatchSource:0}: Error finding container 3f58a0dc2bf54e80a364bae0685d8f8e7bf4412727f6ed72b63ba6e0137ca0f4: Status 404 returned error can't find the container with id 3f58a0dc2bf54e80a364bae0685d8f8e7bf4412727f6ed72b63ba6e0137ca0f4 Jul 05 19:43:00.361597 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:00.361580 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 05 19:43:00.368926 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:00.368901 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:43:01.261011 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:01.260966 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b5b7fd64f-vsx9f" event={"ID":"0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2","Type":"ContainerStarted","Data":"b535340d831f6944367bbf616e45552feb895d8ca120b0f518b4c45823330977"} Jul 05 19:43:01.261507 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:01.261020 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b5b7fd64f-vsx9f" event={"ID":"0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2","Type":"ContainerStarted","Data":"3f58a0dc2bf54e80a364bae0685d8f8e7bf4412727f6ed72b63ba6e0137ca0f4"} Jul 05 19:43:01.280119 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:01.280066 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6b5b7fd64f-vsx9f" podStartSLOduration=1.872836867 podStartE2EDuration="2.28002547s" totalImagesPullingTime="407.188603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:42:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:43:01.278526217 +0000 UTC m=+782.278544327" watchObservedRunningTime="2026-07-05 19:43:01.28002547 +0000 UTC m=+782.280043574" Jul 05 19:43:01.315168 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:01.315132 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78b8656c68-zhw78"] Jul 05 19:43:01.315395 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:01.315365 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-78b8656c68-zhw78" podUID="e2850f87-838b-48a5-a7fd-d0790268e82e" containerName="authorino" containerID="cri-o://dd348a2a42086cd102f33219666a19bcd8f76889d1aae653ba474872c8dd99d4" gracePeriod=30 Jul 05 19:43:01.401892 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:01.401859 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:43:01.506377 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:01.506345 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:43:01.571191 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:01.571164 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78b8656c68-zhw78" Jul 05 19:43:01.714335 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:01.714301 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w68nh\" (UniqueName: \"kubernetes.io/projected/e2850f87-838b-48a5-a7fd-d0790268e82e-kube-api-access-w68nh\") pod \"e2850f87-838b-48a5-a7fd-d0790268e82e\" (UID: \"e2850f87-838b-48a5-a7fd-d0790268e82e\") " Jul 05 19:43:01.714511 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:01.714346 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2850f87-838b-48a5-a7fd-d0790268e82e-tls-cert\") pod \"e2850f87-838b-48a5-a7fd-d0790268e82e\" (UID: \"e2850f87-838b-48a5-a7fd-d0790268e82e\") " Jul 05 19:43:01.716585 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:01.716557 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2850f87-838b-48a5-a7fd-d0790268e82e-kube-api-access-w68nh" (OuterVolumeSpecName: "kube-api-access-w68nh") pod "e2850f87-838b-48a5-a7fd-d0790268e82e" (UID: "e2850f87-838b-48a5-a7fd-d0790268e82e"). InnerVolumeSpecName "kube-api-access-w68nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:43:01.727349 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:01.727312 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2850f87-838b-48a5-a7fd-d0790268e82e-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e2850f87-838b-48a5-a7fd-d0790268e82e" (UID: "e2850f87-838b-48a5-a7fd-d0790268e82e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:43:01.816076 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:01.815929 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w68nh\" (UniqueName: \"kubernetes.io/projected/e2850f87-838b-48a5-a7fd-d0790268e82e-kube-api-access-w68nh\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:43:01.816076 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:01.815958 2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2850f87-838b-48a5-a7fd-d0790268e82e-tls-cert\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:43:02.266861 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:02.266815 2573 generic.go:356] "Generic (PLEG): container finished" podID="e2850f87-838b-48a5-a7fd-d0790268e82e" containerID="dd348a2a42086cd102f33219666a19bcd8f76889d1aae653ba474872c8dd99d4" exitCode=0 Jul 05 19:43:02.267397 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:02.266875 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78b8656c68-zhw78" Jul 05 19:43:02.267397 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:02.266875 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78b8656c68-zhw78" event={"ID":"e2850f87-838b-48a5-a7fd-d0790268e82e","Type":"ContainerDied","Data":"dd348a2a42086cd102f33219666a19bcd8f76889d1aae653ba474872c8dd99d4"} Jul 05 19:43:02.267397 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:02.266933 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78b8656c68-zhw78" event={"ID":"e2850f87-838b-48a5-a7fd-d0790268e82e","Type":"ContainerDied","Data":"acbfa4a3b611a6fd9391bd591a3d6c7876aba92490c01121cc9ac3304190f0fa"} Jul 05 19:43:02.267397 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:02.266959 2573 scope.go:117] "RemoveContainer" containerID="dd348a2a42086cd102f33219666a19bcd8f76889d1aae653ba474872c8dd99d4" Jul 05 19:43:02.275835 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:02.275814 2573 scope.go:117] "RemoveContainer" containerID="dd348a2a42086cd102f33219666a19bcd8f76889d1aae653ba474872c8dd99d4" Jul 05 19:43:02.276153 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:43:02.276130 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd348a2a42086cd102f33219666a19bcd8f76889d1aae653ba474872c8dd99d4\": container with ID starting with dd348a2a42086cd102f33219666a19bcd8f76889d1aae653ba474872c8dd99d4 not found: ID does not exist" containerID="dd348a2a42086cd102f33219666a19bcd8f76889d1aae653ba474872c8dd99d4" Jul 05 19:43:02.276209 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:02.276168 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd348a2a42086cd102f33219666a19bcd8f76889d1aae653ba474872c8dd99d4"} err="failed to get container status \"dd348a2a42086cd102f33219666a19bcd8f76889d1aae653ba474872c8dd99d4\": rpc error: code = NotFound desc = could not find container \"dd348a2a42086cd102f33219666a19bcd8f76889d1aae653ba474872c8dd99d4\": container with ID starting with dd348a2a42086cd102f33219666a19bcd8f76889d1aae653ba474872c8dd99d4 not found: ID does not exist" Jul 05 19:43:02.296014 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:02.295986 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78b8656c68-zhw78"] Jul 05 19:43:02.299724 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:02.299700 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-78b8656c68-zhw78"] Jul 05 19:43:03.476448 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:43:03.476418 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2850f87-838b-48a5-a7fd-d0790268e82e" path="/var/lib/kubelet/pods/e2850f87-838b-48a5-a7fd-d0790268e82e/volumes" Jul 05 19:44:10.949572 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:10.949536 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:44:11.057229 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.057190 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f64fbc48c-t6plz"] Jul 05 19:44:11.057472 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.057446 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-6f64fbc48c-t6plz" podUID="f8a3885c-9f3f-43dd-b156-046739fc7e16" containerName="manager" containerID="cri-o://75caf57a3ad63d8738829055f2fa6845c53ed3515fd5d9093b54a864be27ee77" gracePeriod=30 Jul 05 19:44:11.323551 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.323522 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f64fbc48c-t6plz" Jul 05 19:44:11.353800 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.353770 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz4qm\" (UniqueName: \"kubernetes.io/projected/f8a3885c-9f3f-43dd-b156-046739fc7e16-kube-api-access-fz4qm\") pod \"f8a3885c-9f3f-43dd-b156-046739fc7e16\" (UID: \"f8a3885c-9f3f-43dd-b156-046739fc7e16\") " Jul 05 19:44:11.353982 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.353835 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8a3885c-9f3f-43dd-b156-046739fc7e16-webhook-cert\") pod \"f8a3885c-9f3f-43dd-b156-046739fc7e16\" (UID: \"f8a3885c-9f3f-43dd-b156-046739fc7e16\") " Jul 05 19:44:11.356239 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.356186 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a3885c-9f3f-43dd-b156-046739fc7e16-kube-api-access-fz4qm" (OuterVolumeSpecName: "kube-api-access-fz4qm") pod "f8a3885c-9f3f-43dd-b156-046739fc7e16" (UID: "f8a3885c-9f3f-43dd-b156-046739fc7e16"). InnerVolumeSpecName "kube-api-access-fz4qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:44:11.356434 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.356406 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a3885c-9f3f-43dd-b156-046739fc7e16-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f8a3885c-9f3f-43dd-b156-046739fc7e16" (UID: "f8a3885c-9f3f-43dd-b156-046739fc7e16"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:44:11.455381 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.455349 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f8a3885c-9f3f-43dd-b156-046739fc7e16-webhook-cert\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:44:11.455381 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.455380 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz4qm\" (UniqueName: \"kubernetes.io/projected/f8a3885c-9f3f-43dd-b156-046739fc7e16-kube-api-access-fz4qm\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:44:11.604025 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.603989 2573 generic.go:356] "Generic (PLEG): container finished" podID="f8a3885c-9f3f-43dd-b156-046739fc7e16" containerID="75caf57a3ad63d8738829055f2fa6845c53ed3515fd5d9093b54a864be27ee77" exitCode=0 Jul 05 19:44:11.604250 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.604064 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f64fbc48c-t6plz" Jul 05 19:44:11.604250 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.604072 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f64fbc48c-t6plz" event={"ID":"f8a3885c-9f3f-43dd-b156-046739fc7e16","Type":"ContainerDied","Data":"75caf57a3ad63d8738829055f2fa6845c53ed3515fd5d9093b54a864be27ee77"} Jul 05 19:44:11.604250 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.604141 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f64fbc48c-t6plz" event={"ID":"f8a3885c-9f3f-43dd-b156-046739fc7e16","Type":"ContainerDied","Data":"0f097fe5064fe885b5af7dcda8955654a44522c326de7cc606aa59704794c22d"} Jul 05 19:44:11.604250 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.604163 2573 scope.go:117] "RemoveContainer" containerID="75caf57a3ad63d8738829055f2fa6845c53ed3515fd5d9093b54a864be27ee77" Jul 05 19:44:11.613128 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.613112 2573 scope.go:117] "RemoveContainer" containerID="75caf57a3ad63d8738829055f2fa6845c53ed3515fd5d9093b54a864be27ee77" Jul 05 19:44:11.613373 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:44:11.613355 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75caf57a3ad63d8738829055f2fa6845c53ed3515fd5d9093b54a864be27ee77\": container with ID starting with 75caf57a3ad63d8738829055f2fa6845c53ed3515fd5d9093b54a864be27ee77 not found: ID does not exist" containerID="75caf57a3ad63d8738829055f2fa6845c53ed3515fd5d9093b54a864be27ee77" Jul 05 19:44:11.613419 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.613381 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75caf57a3ad63d8738829055f2fa6845c53ed3515fd5d9093b54a864be27ee77"} err="failed to get container status \"75caf57a3ad63d8738829055f2fa6845c53ed3515fd5d9093b54a864be27ee77\": rpc error: code = NotFound desc = could not find container \"75caf57a3ad63d8738829055f2fa6845c53ed3515fd5d9093b54a864be27ee77\": container with ID starting with 75caf57a3ad63d8738829055f2fa6845c53ed3515fd5d9093b54a864be27ee77 not found: ID does not exist" Jul 05 19:44:11.620132 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.620110 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f64fbc48c-t6plz"] Jul 05 19:44:11.623546 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:11.623526 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f64fbc48c-t6plz"] Jul 05 19:44:12.250854 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.250811 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:44:12.363763 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.363734 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f64fbc48c-xbsjd"] Jul 05 19:44:12.364169 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.364157 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a3885c-9f3f-43dd-b156-046739fc7e16" containerName="manager" Jul 05 19:44:12.364218 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.364170 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a3885c-9f3f-43dd-b156-046739fc7e16" containerName="manager" Jul 05 19:44:12.364218 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.364180 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2850f87-838b-48a5-a7fd-d0790268e82e" containerName="authorino" Jul 05 19:44:12.364218 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.364185 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2850f87-838b-48a5-a7fd-d0790268e82e" containerName="authorino" Jul 05 19:44:12.364311 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.364236 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a3885c-9f3f-43dd-b156-046739fc7e16" containerName="manager" Jul 05 19:44:12.364311 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.364247 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2850f87-838b-48a5-a7fd-d0790268e82e" containerName="authorino" Jul 05 19:44:12.368573 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.368557 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f64fbc48c-xbsjd" Jul 05 19:44:12.372853 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.372828 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 05 19:44:12.373034 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.372828 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 05 19:44:12.373178 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.373152 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-c2849\"" Jul 05 19:44:12.375214 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.375193 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f64fbc48c-xbsjd"] Jul 05 19:44:12.465970 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.465931 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ec720cb-1b72-4eff-b8f7-059e0be3ea12-webhook-cert\") pod \"maas-controller-6f64fbc48c-xbsjd\" (UID: \"4ec720cb-1b72-4eff-b8f7-059e0be3ea12\") " pod="opendatahub/maas-controller-6f64fbc48c-xbsjd" Jul 05 19:44:12.466181 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.466015 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfkcd\" (UniqueName: \"kubernetes.io/projected/4ec720cb-1b72-4eff-b8f7-059e0be3ea12-kube-api-access-qfkcd\") pod \"maas-controller-6f64fbc48c-xbsjd\" (UID: \"4ec720cb-1b72-4eff-b8f7-059e0be3ea12\") " pod="opendatahub/maas-controller-6f64fbc48c-xbsjd" Jul 05 19:44:12.566721 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.566631 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfkcd\" (UniqueName: \"kubernetes.io/projected/4ec720cb-1b72-4eff-b8f7-059e0be3ea12-kube-api-access-qfkcd\") pod \"maas-controller-6f64fbc48c-xbsjd\" (UID: \"4ec720cb-1b72-4eff-b8f7-059e0be3ea12\") " pod="opendatahub/maas-controller-6f64fbc48c-xbsjd" Jul 05 19:44:12.566721 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.566700 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ec720cb-1b72-4eff-b8f7-059e0be3ea12-webhook-cert\") pod \"maas-controller-6f64fbc48c-xbsjd\" (UID: \"4ec720cb-1b72-4eff-b8f7-059e0be3ea12\") " pod="opendatahub/maas-controller-6f64fbc48c-xbsjd" Jul 05 19:44:12.569252 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.569234 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4ec720cb-1b72-4eff-b8f7-059e0be3ea12-webhook-cert\") pod \"maas-controller-6f64fbc48c-xbsjd\" (UID: \"4ec720cb-1b72-4eff-b8f7-059e0be3ea12\") " pod="opendatahub/maas-controller-6f64fbc48c-xbsjd" Jul 05 19:44:12.574884 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.574860 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfkcd\" (UniqueName: \"kubernetes.io/projected/4ec720cb-1b72-4eff-b8f7-059e0be3ea12-kube-api-access-qfkcd\") pod \"maas-controller-6f64fbc48c-xbsjd\" (UID: \"4ec720cb-1b72-4eff-b8f7-059e0be3ea12\") " pod="opendatahub/maas-controller-6f64fbc48c-xbsjd" Jul 05 19:44:12.680030 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.679999 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f64fbc48c-xbsjd" Jul 05 19:44:12.812326 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:12.812301 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f64fbc48c-xbsjd"] Jul 05 19:44:12.814224 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:44:12.814196 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ec720cb_1b72_4eff_b8f7_059e0be3ea12.slice/crio-af982189c42afeb2e2bc9e7b3255755ce16bba38e585ca5c4cf8cd530c30660c WatchSource:0}: Error finding container af982189c42afeb2e2bc9e7b3255755ce16bba38e585ca5c4cf8cd530c30660c: Status 404 returned error can't find the container with id af982189c42afeb2e2bc9e7b3255755ce16bba38e585ca5c4cf8cd530c30660c Jul 05 19:44:13.469895 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:13.469855 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a3885c-9f3f-43dd-b156-046739fc7e16" path="/var/lib/kubelet/pods/f8a3885c-9f3f-43dd-b156-046739fc7e16/volumes" Jul 05 19:44:13.615846 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:13.615744 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f64fbc48c-xbsjd" event={"ID":"4ec720cb-1b72-4eff-b8f7-059e0be3ea12","Type":"ContainerStarted","Data":"fbe3c7b8269ce1c6e6b27de37076a8599f26154521ec4e37af3c7f99084c0628"} Jul 05 19:44:13.615846 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:13.615807 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f64fbc48c-xbsjd" event={"ID":"4ec720cb-1b72-4eff-b8f7-059e0be3ea12","Type":"ContainerStarted","Data":"af982189c42afeb2e2bc9e7b3255755ce16bba38e585ca5c4cf8cd530c30660c"} Jul 05 19:44:13.615846 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:13.615843 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f64fbc48c-xbsjd" Jul 05 19:44:13.632571 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:13.632525 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f64fbc48c-xbsjd" podStartSLOduration=1.132715927 podStartE2EDuration="1.632509829s" totalImagesPullingTime="499.793902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:44:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:44:13.630450064 +0000 UTC m=+854.630468169" watchObservedRunningTime="2026-07-05 19:44:13.632509829 +0000 UTC m=+854.632527933" Jul 05 19:44:24.627538 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:24.627505 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f64fbc48c-xbsjd" Jul 05 19:44:47.391143 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:47.391099 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:44:54.650654 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:54.650572 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:44:54.750510 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:54.750472 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:44:54.868593 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:54.868560 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:44:59.482738 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:59.482708 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 19:44:59.483211 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:44:59.483179 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 19:45:00.141758 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:00.141724 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29721345-74rpw"] Jul 05 19:45:00.146495 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:00.146478 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29721345-74rpw" Jul 05 19:45:00.151439 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:00.151419 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29721345-74rpw"] Jul 05 19:45:00.208391 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:00.208350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzbbd\" (UniqueName: \"kubernetes.io/projected/b4de75db-4fed-4039-ad76-93383a7c5052-kube-api-access-fzbbd\") pod \"maas-api-key-cleanup-29721345-74rpw\" (UID: \"b4de75db-4fed-4039-ad76-93383a7c5052\") " pod="opendatahub/maas-api-key-cleanup-29721345-74rpw" Jul 05 19:45:00.308885 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:00.308849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzbbd\" (UniqueName: \"kubernetes.io/projected/b4de75db-4fed-4039-ad76-93383a7c5052-kube-api-access-fzbbd\") pod \"maas-api-key-cleanup-29721345-74rpw\" (UID: \"b4de75db-4fed-4039-ad76-93383a7c5052\") " pod="opendatahub/maas-api-key-cleanup-29721345-74rpw" Jul 05 19:45:00.316779 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:00.316747 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzbbd\" (UniqueName: \"kubernetes.io/projected/b4de75db-4fed-4039-ad76-93383a7c5052-kube-api-access-fzbbd\") pod \"maas-api-key-cleanup-29721345-74rpw\" (UID: \"b4de75db-4fed-4039-ad76-93383a7c5052\") " pod="opendatahub/maas-api-key-cleanup-29721345-74rpw" Jul 05 19:45:00.409732 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:00.409644 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:45:00.458436 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:00.458402 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29721345-74rpw" Jul 05 19:45:00.521236 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:00.521205 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:45:00.615302 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:00.615278 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29721345-74rpw"] Jul 05 19:45:00.617313 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:45:00.617287 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4de75db_4fed_4039_ad76_93383a7c5052.slice/crio-d94343d12c3a31e627f7d87fc4f437f7bd61bf509abf7ff155d831fb20aec141 WatchSource:0}: Error finding container d94343d12c3a31e627f7d87fc4f437f7bd61bf509abf7ff155d831fb20aec141: Status 404 returned error can't find the container with id d94343d12c3a31e627f7d87fc4f437f7bd61bf509abf7ff155d831fb20aec141 Jul 05 19:45:00.636818 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:00.636783 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:45:00.757997 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:00.757957 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:45:00.837624 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:00.837594 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29721345-74rpw" event={"ID":"b4de75db-4fed-4039-ad76-93383a7c5052","Type":"ContainerStarted","Data":"d94343d12c3a31e627f7d87fc4f437f7bd61bf509abf7ff155d831fb20aec141"} Jul 05 19:45:03.855270 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:03.855233 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29721345-74rpw" event={"ID":"b4de75db-4fed-4039-ad76-93383a7c5052","Type":"ContainerStarted","Data":"8e1e2f11f8c0287887c33d7695097ca08bcab258098799cfb7ee01b8ed016118"} Jul 05 19:45:03.872701 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:03.872650 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29721345-74rpw" podStartSLOduration=1.349449514 podStartE2EDuration="3.872631634s" totalImagesPullingTime="2.52318212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:45:03.870166289 +0000 UTC m=+904.870184409" watchObservedRunningTime="2026-07-05 19:45:03.872631634 +0000 UTC m=+904.872649738" Jul 05 19:45:04.864633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:04.864596 2573 generic.go:356] "Generic (PLEG): container finished" podID="b4de75db-4fed-4039-ad76-93383a7c5052" containerID="8e1e2f11f8c0287887c33d7695097ca08bcab258098799cfb7ee01b8ed016118" exitCode=0 Jul 05 19:45:04.865097 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:04.864685 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29721345-74rpw" event={"ID":"b4de75db-4fed-4039-ad76-93383a7c5052","Type":"ContainerDied","Data":"8e1e2f11f8c0287887c33d7695097ca08bcab258098799cfb7ee01b8ed016118"} Jul 05 19:45:05.999172 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:05.999148 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29721345-74rpw" Jul 05 19:45:06.068304 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:06.068266 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzbbd\" (UniqueName: \"kubernetes.io/projected/b4de75db-4fed-4039-ad76-93383a7c5052-kube-api-access-fzbbd\") pod \"b4de75db-4fed-4039-ad76-93383a7c5052\" (UID: \"b4de75db-4fed-4039-ad76-93383a7c5052\") " Jul 05 19:45:06.070443 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:06.070418 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4de75db-4fed-4039-ad76-93383a7c5052-kube-api-access-fzbbd" (OuterVolumeSpecName: "kube-api-access-fzbbd") pod "b4de75db-4fed-4039-ad76-93383a7c5052" (UID: "b4de75db-4fed-4039-ad76-93383a7c5052"). InnerVolumeSpecName "kube-api-access-fzbbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:45:06.169148 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:06.169029 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzbbd\" (UniqueName: \"kubernetes.io/projected/b4de75db-4fed-4039-ad76-93383a7c5052-kube-api-access-fzbbd\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:45:06.403969 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:06.403937 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:45:06.874809 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:06.874779 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29721345-74rpw" Jul 05 19:45:06.874985 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:06.874815 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29721345-74rpw" event={"ID":"b4de75db-4fed-4039-ad76-93383a7c5052","Type":"ContainerDied","Data":"d94343d12c3a31e627f7d87fc4f437f7bd61bf509abf7ff155d831fb20aec141"} Jul 05 19:45:06.874985 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:06.874863 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d94343d12c3a31e627f7d87fc4f437f7bd61bf509abf7ff155d831fb20aec141" Jul 05 19:45:07.520361 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:07.520328 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:45:49.790748 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:49.790707 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:45:49.887738 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:45:49.887692 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:47:40.057979 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:47:40.057939 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:49:59.513071 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:49:59.513022 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 19:49:59.515830 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:49:59.515810 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 19:53:45.266568 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:53:45.266519 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:53:52.400204 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:53:52.400123 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:54:59.548401 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:54:59.548365 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 19:54:59.552109 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:54:59.552083 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 19:55:38.565169 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:55:38.565076 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:55:45.339831 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:55:45.339784 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:55:53.585817 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:55:53.585770 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:55:58.673793 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:55:58.673748 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:01.436427 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:01.436387 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:01.533006 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:01.532963 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:01.863440 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:01.863403 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:07.592228 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:07.592181 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:07.651209 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:07.651159 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:07.880584 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:07.880541 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:07.976821 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:07.976789 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:09.049737 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.049706 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:09.128737 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.128685 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj"] Jul 05 19:56:09.129415 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.129384 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4de75db-4fed-4039-ad76-93383a7c5052" containerName="cleanup" Jul 05 19:56:09.129415 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.129416 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4de75db-4fed-4039-ad76-93383a7c5052" containerName="cleanup" Jul 05 19:56:09.129605 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.129520 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4de75db-4fed-4039-ad76-93383a7c5052" containerName="cleanup" Jul 05 19:56:09.130461 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.130433 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:09.133152 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.133132 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" Jul 05 19:56:09.135914 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.135892 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-5fe4b92a-serving-cert\"" Jul 05 19:56:09.144001 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.143967 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj"] Jul 05 19:56:09.194546 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.194515 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:09.226860 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.226818 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr9ms\" (UniqueName: \"kubernetes.io/projected/67b7b24a-19a2-43d5-84b9-98096be5569f-kube-api-access-lr9ms\") pod \"maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj\" (UID: \"67b7b24a-19a2-43d5-84b9-98096be5569f\") " pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" Jul 05 19:56:09.227076 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.226891 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b7b24a-19a2-43d5-84b9-98096be5569f-maas-api-tls\") pod \"maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj\" (UID: \"67b7b24a-19a2-43d5-84b9-98096be5569f\") " pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" Jul 05 19:56:09.227076 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.226916 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b7b24a-19a2-43d5-84b9-98096be5569f-openshift-service-ca\") pod \"maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj\" (UID: \"67b7b24a-19a2-43d5-84b9-98096be5569f\") " pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" Jul 05 19:56:09.327997 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.327898 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b7b24a-19a2-43d5-84b9-98096be5569f-maas-api-tls\") pod \"maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj\" (UID: \"67b7b24a-19a2-43d5-84b9-98096be5569f\") " pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" Jul 05 19:56:09.327997 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.327941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b7b24a-19a2-43d5-84b9-98096be5569f-openshift-service-ca\") pod \"maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj\" (UID: \"67b7b24a-19a2-43d5-84b9-98096be5569f\") " pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" Jul 05 19:56:09.328269 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:56:09.328084 2573 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-5fe4b92a-serving-cert: secret "maas-api-e2e-ait-5fe4b92a-serving-cert" not found Jul 05 19:56:09.328269 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.328096 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr9ms\" (UniqueName: \"kubernetes.io/projected/67b7b24a-19a2-43d5-84b9-98096be5569f-kube-api-access-lr9ms\") pod \"maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj\" (UID: \"67b7b24a-19a2-43d5-84b9-98096be5569f\") " pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" Jul 05 19:56:09.328269 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:56:09.328172 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b7b24a-19a2-43d5-84b9-98096be5569f-maas-api-tls podName:67b7b24a-19a2-43d5-84b9-98096be5569f nodeName:}" failed. No retries permitted until 2026-07-05 19:56:09.828150466 +0000 UTC m=+1570.828168560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/67b7b24a-19a2-43d5-84b9-98096be5569f-maas-api-tls") pod "maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" (UID: "67b7b24a-19a2-43d5-84b9-98096be5569f") : secret "maas-api-e2e-ait-5fe4b92a-serving-cert" not found Jul 05 19:56:09.328643 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.328624 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b7b24a-19a2-43d5-84b9-98096be5569f-openshift-service-ca\") pod \"maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj\" (UID: \"67b7b24a-19a2-43d5-84b9-98096be5569f\") " pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" Jul 05 19:56:09.337743 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.337707 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr9ms\" (UniqueName: \"kubernetes.io/projected/67b7b24a-19a2-43d5-84b9-98096be5569f-kube-api-access-lr9ms\") pod \"maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj\" (UID: \"67b7b24a-19a2-43d5-84b9-98096be5569f\") " pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" Jul 05 19:56:09.447645 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.447609 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:09.579296 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.579189 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:09.679529 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.679474 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:09.831523 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.831421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b7b24a-19a2-43d5-84b9-98096be5569f-maas-api-tls\") pod \"maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj\" (UID: \"67b7b24a-19a2-43d5-84b9-98096be5569f\") " pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" Jul 05 19:56:09.834449 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:09.834417 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b7b24a-19a2-43d5-84b9-98096be5569f-maas-api-tls\") pod \"maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj\" (UID: \"67b7b24a-19a2-43d5-84b9-98096be5569f\") " pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" Jul 05 19:56:10.047901 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:10.047862 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" Jul 05 19:56:10.203857 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:10.203830 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj"] Jul 05 19:56:10.206267 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:56:10.206232 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b7b24a_19a2_43d5_84b9_98096be5569f.slice/crio-32c81fabb9abde8ca79bf7a0b2f6494788ff2e1efba82c3cd051cf50cbf73c4e WatchSource:0}: Error finding container 32c81fabb9abde8ca79bf7a0b2f6494788ff2e1efba82c3cd051cf50cbf73c4e: Status 404 returned error can't find the container with id 32c81fabb9abde8ca79bf7a0b2f6494788ff2e1efba82c3cd051cf50cbf73c4e Jul 05 19:56:10.207665 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:10.207642 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 05 19:56:11.178487 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:11.178453 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" event={"ID":"67b7b24a-19a2-43d5-84b9-98096be5569f","Type":"ContainerStarted","Data":"10973a655ba50c390e6ca152719bf8e2377109b32fa4ba81ba5a52053b383e92"} Jul 05 19:56:11.178656 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:11.178504 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" event={"ID":"67b7b24a-19a2-43d5-84b9-98096be5569f","Type":"ContainerStarted","Data":"32c81fabb9abde8ca79bf7a0b2f6494788ff2e1efba82c3cd051cf50cbf73c4e"} Jul 05 19:56:11.178656 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:11.178573 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" Jul 05 19:56:11.197315 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:11.197259 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" podStartSLOduration=1.7867946 podStartE2EDuration="2.19724099s" totalImagesPullingTime="410.44639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:56:11.194799557 +0000 UTC m=+1572.194817664" watchObservedRunningTime="2026-07-05 19:56:11.19724099 +0000 UTC m=+1572.197259116" Jul 05 19:56:11.335729 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:11.335688 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:14.056659 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.056624 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:14.239221 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.239182 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:14.298739 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.298696 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj"] Jul 05 19:56:14.299067 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.299015 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" podUID="67b7b24a-19a2-43d5-84b9-98096be5569f" containerName="maas-api" containerID="cri-o://10973a655ba50c390e6ca152719bf8e2377109b32fa4ba81ba5a52053b383e92" gracePeriod=30 Jul 05 19:56:14.319882 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.319852 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:14.420738 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.420706 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:14.553021 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.552988 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" Jul 05 19:56:14.677270 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.677159 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b7b24a-19a2-43d5-84b9-98096be5569f-maas-api-tls\") pod \"67b7b24a-19a2-43d5-84b9-98096be5569f\" (UID: \"67b7b24a-19a2-43d5-84b9-98096be5569f\") " Jul 05 19:56:14.677270 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.677257 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr9ms\" (UniqueName: \"kubernetes.io/projected/67b7b24a-19a2-43d5-84b9-98096be5569f-kube-api-access-lr9ms\") pod \"67b7b24a-19a2-43d5-84b9-98096be5569f\" (UID: \"67b7b24a-19a2-43d5-84b9-98096be5569f\") " Jul 05 19:56:14.677559 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.677344 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b7b24a-19a2-43d5-84b9-98096be5569f-openshift-service-ca\") pod \"67b7b24a-19a2-43d5-84b9-98096be5569f\" (UID: \"67b7b24a-19a2-43d5-84b9-98096be5569f\") " Jul 05 19:56:14.677754 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.677727 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b7b24a-19a2-43d5-84b9-98096be5569f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67b7b24a-19a2-43d5-84b9-98096be5569f" (UID: "67b7b24a-19a2-43d5-84b9-98096be5569f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:56:14.679550 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.679527 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b7b24a-19a2-43d5-84b9-98096be5569f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67b7b24a-19a2-43d5-84b9-98096be5569f" (UID: "67b7b24a-19a2-43d5-84b9-98096be5569f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:56:14.679651 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.679614 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b7b24a-19a2-43d5-84b9-98096be5569f-kube-api-access-lr9ms" (OuterVolumeSpecName: "kube-api-access-lr9ms") pod "67b7b24a-19a2-43d5-84b9-98096be5569f" (UID: "67b7b24a-19a2-43d5-84b9-98096be5569f"). InnerVolumeSpecName "kube-api-access-lr9ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:56:14.778638 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.778591 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b7b24a-19a2-43d5-84b9-98096be5569f-openshift-service-ca\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:56:14.778638 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.778630 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b7b24a-19a2-43d5-84b9-98096be5569f-maas-api-tls\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:56:14.778638 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:14.778645 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr9ms\" (UniqueName: \"kubernetes.io/projected/67b7b24a-19a2-43d5-84b9-98096be5569f-kube-api-access-lr9ms\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:56:15.201152 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:15.201113 2573 generic.go:356] "Generic (PLEG): container finished" podID="67b7b24a-19a2-43d5-84b9-98096be5569f" containerID="10973a655ba50c390e6ca152719bf8e2377109b32fa4ba81ba5a52053b383e92" exitCode=0 Jul 05 19:56:15.201152 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:15.201154 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" event={"ID":"67b7b24a-19a2-43d5-84b9-98096be5569f","Type":"ContainerDied","Data":"10973a655ba50c390e6ca152719bf8e2377109b32fa4ba81ba5a52053b383e92"} Jul 05 19:56:15.201643 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:15.201163 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" Jul 05 19:56:15.201643 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:15.201201 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj" event={"ID":"67b7b24a-19a2-43d5-84b9-98096be5569f","Type":"ContainerDied","Data":"32c81fabb9abde8ca79bf7a0b2f6494788ff2e1efba82c3cd051cf50cbf73c4e"} Jul 05 19:56:15.201643 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:15.201226 2573 scope.go:117] "RemoveContainer" containerID="10973a655ba50c390e6ca152719bf8e2377109b32fa4ba81ba5a52053b383e92" Jul 05 19:56:15.211607 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:15.211581 2573 scope.go:117] "RemoveContainer" containerID="10973a655ba50c390e6ca152719bf8e2377109b32fa4ba81ba5a52053b383e92" Jul 05 19:56:15.211898 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:56:15.211881 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10973a655ba50c390e6ca152719bf8e2377109b32fa4ba81ba5a52053b383e92\": container with ID starting with 10973a655ba50c390e6ca152719bf8e2377109b32fa4ba81ba5a52053b383e92 not found: ID does not exist" containerID="10973a655ba50c390e6ca152719bf8e2377109b32fa4ba81ba5a52053b383e92" Jul 05 19:56:15.211938 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:15.211907 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10973a655ba50c390e6ca152719bf8e2377109b32fa4ba81ba5a52053b383e92"} err="failed to get container status \"10973a655ba50c390e6ca152719bf8e2377109b32fa4ba81ba5a52053b383e92\": rpc error: code = NotFound desc = could not find container \"10973a655ba50c390e6ca152719bf8e2377109b32fa4ba81ba5a52053b383e92\": container with ID starting with 10973a655ba50c390e6ca152719bf8e2377109b32fa4ba81ba5a52053b383e92 not found: ID does not exist" Jul 05 19:56:15.222406 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:15.222368 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj"] Jul 05 19:56:15.226061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:15.226017 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-5fe4b92a-58ccf5ffbf-tp2sj"] Jul 05 19:56:15.470159 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:15.470026 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67b7b24a-19a2-43d5-84b9-98096be5569f" path="/var/lib/kubelet/pods/67b7b24a-19a2-43d5-84b9-98096be5569f/volumes" Jul 05 19:56:16.046956 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:16.046915 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:20.081392 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:20.081352 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:20.139348 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:20.139309 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:20.289714 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:20.289679 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:20.375019 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:20.374917 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:21.468668 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.468625 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:21.573944 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.573898 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:21.593816 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.593778 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx"] Jul 05 19:56:21.594297 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.594268 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b7b24a-19a2-43d5-84b9-98096be5569f" containerName="maas-api" Jul 05 19:56:21.594297 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.594299 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b7b24a-19a2-43d5-84b9-98096be5569f" containerName="maas-api" Jul 05 19:56:21.594409 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.594375 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b7b24a-19a2-43d5-84b9-98096be5569f" containerName="maas-api" Jul 05 19:56:21.599115 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.599086 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" Jul 05 19:56:21.602633 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.602603 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-ebeca618-serving-cert\"" Jul 05 19:56:21.617888 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.617855 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx"] Jul 05 19:56:21.652084 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.652023 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd889ec-becd-4b83-9441-324671e327ae-openshift-service-ca\") pod \"maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx\" (UID: \"ebd889ec-becd-4b83-9441-324671e327ae\") " pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" Jul 05 19:56:21.652316 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.652235 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd889ec-becd-4b83-9441-324671e327ae-maas-api-tls\") pod \"maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx\" (UID: \"ebd889ec-becd-4b83-9441-324671e327ae\") " pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" Jul 05 19:56:21.652316 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.652281 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfzpj\" (UniqueName: \"kubernetes.io/projected/ebd889ec-becd-4b83-9441-324671e327ae-kube-api-access-lfzpj\") pod \"maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx\" (UID: \"ebd889ec-becd-4b83-9441-324671e327ae\") " pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" Jul 05 19:56:21.683675 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.683635 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:21.753116 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.753078 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd889ec-becd-4b83-9441-324671e327ae-openshift-service-ca\") pod \"maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx\" (UID: \"ebd889ec-becd-4b83-9441-324671e327ae\") " pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" Jul 05 19:56:21.753316 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.753191 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd889ec-becd-4b83-9441-324671e327ae-maas-api-tls\") pod \"maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx\" (UID: \"ebd889ec-becd-4b83-9441-324671e327ae\") " pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" Jul 05 19:56:21.753316 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.753215 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfzpj\" (UniqueName: \"kubernetes.io/projected/ebd889ec-becd-4b83-9441-324671e327ae-kube-api-access-lfzpj\") pod \"maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx\" (UID: \"ebd889ec-becd-4b83-9441-324671e327ae\") " pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" Jul 05 19:56:21.753394 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:56:21.753339 2573 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-ebeca618-serving-cert: secret "maas-api-e2e-ait-ebeca618-serving-cert" not found Jul 05 19:56:21.753434 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:56:21.753425 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebd889ec-becd-4b83-9441-324671e327ae-maas-api-tls podName:ebd889ec-becd-4b83-9441-324671e327ae nodeName:}" failed. No retries permitted until 2026-07-05 19:56:22.253399719 +0000 UTC m=+1583.253417813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ebd889ec-becd-4b83-9441-324671e327ae-maas-api-tls") pod "maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" (UID: "ebd889ec-becd-4b83-9441-324671e327ae") : secret "maas-api-e2e-ait-ebeca618-serving-cert" not found Jul 05 19:56:21.753730 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.753711 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd889ec-becd-4b83-9441-324671e327ae-openshift-service-ca\") pod \"maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx\" (UID: \"ebd889ec-becd-4b83-9441-324671e327ae\") " pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" Jul 05 19:56:21.762627 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:21.762589 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfzpj\" (UniqueName: \"kubernetes.io/projected/ebd889ec-becd-4b83-9441-324671e327ae-kube-api-access-lfzpj\") pod \"maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx\" (UID: \"ebd889ec-becd-4b83-9441-324671e327ae\") " pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" Jul 05 19:56:22.101689 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:22.101555 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:22.198846 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:22.198801 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:22.258903 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:22.258866 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd889ec-becd-4b83-9441-324671e327ae-maas-api-tls\") pod \"maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx\" (UID: \"ebd889ec-becd-4b83-9441-324671e327ae\") " pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" Jul 05 19:56:22.261636 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:22.261616 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd889ec-becd-4b83-9441-324671e327ae-maas-api-tls\") pod \"maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx\" (UID: \"ebd889ec-becd-4b83-9441-324671e327ae\") " pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" Jul 05 19:56:22.510261 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:22.510200 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" Jul 05 19:56:22.665965 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:22.665927 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx"] Jul 05 19:56:22.668826 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:56:22.668790 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd889ec_becd_4b83_9441_324671e327ae.slice/crio-1f08864b2eb3bc993c7755131c8fbea011cd4e8e2be20c0890ff84314ee0c79e WatchSource:0}: Error finding container 1f08864b2eb3bc993c7755131c8fbea011cd4e8e2be20c0890ff84314ee0c79e: Status 404 returned error can't find the container with id 1f08864b2eb3bc993c7755131c8fbea011cd4e8e2be20c0890ff84314ee0c79e Jul 05 19:56:23.251802 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:23.251759 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" event={"ID":"ebd889ec-becd-4b83-9441-324671e327ae","Type":"ContainerStarted","Data":"23388de5a16f8196ff1b9bb41982dd82d76f50b423a957952d97f8629592ac53"} Jul 05 19:56:23.252020 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:23.251832 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" Jul 05 19:56:23.252020 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:23.251856 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" event={"ID":"ebd889ec-becd-4b83-9441-324671e327ae","Type":"ContainerStarted","Data":"1f08864b2eb3bc993c7755131c8fbea011cd4e8e2be20c0890ff84314ee0c79e"} Jul 05 19:56:23.269755 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:23.269640 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" podStartSLOduration=1.932877873 podStartE2EDuration="2.269619049s" totalImagesPullingTime="336.741176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:56:23.267105052 +0000 UTC m=+1584.267123158" watchObservedRunningTime="2026-07-05 19:56:23.269619049 +0000 UTC m=+1584.269637607" Jul 05 19:56:23.627629 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:23.627596 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:26.590241 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:26.590201 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:26.902312 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:26.902206 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx"] Jul 05 19:56:26.902492 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:26.902452 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" podUID="ebd889ec-becd-4b83-9441-324671e327ae" containerName="maas-api" containerID="cri-o://23388de5a16f8196ff1b9bb41982dd82d76f50b423a957952d97f8629592ac53" gracePeriod=30 Jul 05 19:56:26.924628 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:26.924593 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:27.167077 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.166973 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" Jul 05 19:56:27.205066 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.205012 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd889ec-becd-4b83-9441-324671e327ae-openshift-service-ca\") pod \"ebd889ec-becd-4b83-9441-324671e327ae\" (UID: \"ebd889ec-becd-4b83-9441-324671e327ae\") " Jul 05 19:56:27.205297 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.205087 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfzpj\" (UniqueName: \"kubernetes.io/projected/ebd889ec-becd-4b83-9441-324671e327ae-kube-api-access-lfzpj\") pod \"ebd889ec-becd-4b83-9441-324671e327ae\" (UID: \"ebd889ec-becd-4b83-9441-324671e327ae\") " Jul 05 19:56:27.205297 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.205119 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd889ec-becd-4b83-9441-324671e327ae-maas-api-tls\") pod \"ebd889ec-becd-4b83-9441-324671e327ae\" (UID: \"ebd889ec-becd-4b83-9441-324671e327ae\") " Jul 05 19:56:27.205461 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.205432 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd889ec-becd-4b83-9441-324671e327ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ebd889ec-becd-4b83-9441-324671e327ae" (UID: "ebd889ec-becd-4b83-9441-324671e327ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:56:27.207682 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.207645 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd889ec-becd-4b83-9441-324671e327ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ebd889ec-becd-4b83-9441-324671e327ae" (UID: "ebd889ec-becd-4b83-9441-324671e327ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:56:27.207853 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.207779 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd889ec-becd-4b83-9441-324671e327ae-kube-api-access-lfzpj" (OuterVolumeSpecName: "kube-api-access-lfzpj") pod "ebd889ec-becd-4b83-9441-324671e327ae" (UID: "ebd889ec-becd-4b83-9441-324671e327ae"). InnerVolumeSpecName "kube-api-access-lfzpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:56:27.274009 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.273970 2573 generic.go:356] "Generic (PLEG): container finished" podID="ebd889ec-becd-4b83-9441-324671e327ae" containerID="23388de5a16f8196ff1b9bb41982dd82d76f50b423a957952d97f8629592ac53" exitCode=0 Jul 05 19:56:27.274235 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.274018 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" Jul 05 19:56:27.274235 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.274077 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" event={"ID":"ebd889ec-becd-4b83-9441-324671e327ae","Type":"ContainerDied","Data":"23388de5a16f8196ff1b9bb41982dd82d76f50b423a957952d97f8629592ac53"} Jul 05 19:56:27.274235 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.274143 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx" event={"ID":"ebd889ec-becd-4b83-9441-324671e327ae","Type":"ContainerDied","Data":"1f08864b2eb3bc993c7755131c8fbea011cd4e8e2be20c0890ff84314ee0c79e"} Jul 05 19:56:27.274235 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.274168 2573 scope.go:117] "RemoveContainer" containerID="23388de5a16f8196ff1b9bb41982dd82d76f50b423a957952d97f8629592ac53" Jul 05 19:56:27.285065 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.285028 2573 scope.go:117] "RemoveContainer" containerID="23388de5a16f8196ff1b9bb41982dd82d76f50b423a957952d97f8629592ac53" Jul 05 19:56:27.285388 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:56:27.285363 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23388de5a16f8196ff1b9bb41982dd82d76f50b423a957952d97f8629592ac53\": container with ID starting with 23388de5a16f8196ff1b9bb41982dd82d76f50b423a957952d97f8629592ac53 not found: ID does not exist" containerID="23388de5a16f8196ff1b9bb41982dd82d76f50b423a957952d97f8629592ac53" Jul 05 19:56:27.285479 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.285397 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23388de5a16f8196ff1b9bb41982dd82d76f50b423a957952d97f8629592ac53"} err="failed to get container status \"23388de5a16f8196ff1b9bb41982dd82d76f50b423a957952d97f8629592ac53\": rpc error: code = NotFound desc = could not find container \"23388de5a16f8196ff1b9bb41982dd82d76f50b423a957952d97f8629592ac53\": container with ID starting with 23388de5a16f8196ff1b9bb41982dd82d76f50b423a957952d97f8629592ac53 not found: ID does not exist" Jul 05 19:56:27.296573 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.296533 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx"] Jul 05 19:56:27.300340 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.300303 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-ebeca618-6c8bcf8484-m45kx"] Jul 05 19:56:27.306227 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.306198 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfzpj\" (UniqueName: \"kubernetes.io/projected/ebd889ec-becd-4b83-9441-324671e327ae-kube-api-access-lfzpj\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:56:27.306227 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.306227 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ebd889ec-becd-4b83-9441-324671e327ae-maas-api-tls\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:56:27.306398 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.306242 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd889ec-becd-4b83-9441-324671e327ae-openshift-service-ca\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:56:27.470025 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:27.469939 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd889ec-becd-4b83-9441-324671e327ae" path="/var/lib/kubelet/pods/ebd889ec-becd-4b83-9441-324671e327ae/volumes" Jul 05 19:56:32.875998 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:32.875958 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:32.978790 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:32.978755 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:38.710881 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:38.710841 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:38.809591 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:38.809552 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:38.903053 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:38.902998 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:39.855095 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:39.855065 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:39.906099 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:39.906033 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:40.083291 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.083224 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:40.139402 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.139305 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m"] Jul 05 19:56:40.140736 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.140706 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd889ec-becd-4b83-9441-324671e327ae" containerName="maas-api" Jul 05 19:56:40.140904 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.140892 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd889ec-becd-4b83-9441-324671e327ae" containerName="maas-api" Jul 05 19:56:40.141222 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.141207 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd889ec-becd-4b83-9441-324671e327ae" containerName="maas-api" Jul 05 19:56:40.147655 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.147623 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" Jul 05 19:56:40.153054 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.152999 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-da4a26de-serving-cert\"" Jul 05 19:56:40.153419 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.153393 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m"] Jul 05 19:56:40.155308 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.155254 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:40.218104 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.218065 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:40.237630 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.237591 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/911ec323-a66d-4d26-92b0-92b84d418131-maas-api-tls\") pod \"maas-api-e2e-derive-da4a26de-6657cc8699-25k5m\" (UID: \"911ec323-a66d-4d26-92b0-92b84d418131\") " pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" Jul 05 19:56:40.237787 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.237634 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc7v2\" (UniqueName: \"kubernetes.io/projected/911ec323-a66d-4d26-92b0-92b84d418131-kube-api-access-qc7v2\") pod \"maas-api-e2e-derive-da4a26de-6657cc8699-25k5m\" (UID: \"911ec323-a66d-4d26-92b0-92b84d418131\") " pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" Jul 05 19:56:40.237787 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.237745 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/911ec323-a66d-4d26-92b0-92b84d418131-openshift-service-ca\") pod \"maas-api-e2e-derive-da4a26de-6657cc8699-25k5m\" (UID: \"911ec323-a66d-4d26-92b0-92b84d418131\") " pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" Jul 05 19:56:40.339335 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.339299 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/911ec323-a66d-4d26-92b0-92b84d418131-maas-api-tls\") pod \"maas-api-e2e-derive-da4a26de-6657cc8699-25k5m\" (UID: \"911ec323-a66d-4d26-92b0-92b84d418131\") " pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" Jul 05 19:56:40.339335 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.339337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7v2\" (UniqueName: \"kubernetes.io/projected/911ec323-a66d-4d26-92b0-92b84d418131-kube-api-access-qc7v2\") pod \"maas-api-e2e-derive-da4a26de-6657cc8699-25k5m\" (UID: \"911ec323-a66d-4d26-92b0-92b84d418131\") " pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" Jul 05 19:56:40.339576 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.339402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/911ec323-a66d-4d26-92b0-92b84d418131-openshift-service-ca\") pod \"maas-api-e2e-derive-da4a26de-6657cc8699-25k5m\" (UID: \"911ec323-a66d-4d26-92b0-92b84d418131\") " pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" Jul 05 19:56:40.339576 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:56:40.339452 2573 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-derive-da4a26de-serving-cert: secret "maas-api-e2e-derive-da4a26de-serving-cert" not found Jul 05 19:56:40.339576 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:56:40.339546 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/911ec323-a66d-4d26-92b0-92b84d418131-maas-api-tls podName:911ec323-a66d-4d26-92b0-92b84d418131 nodeName:}" failed. No retries permitted until 2026-07-05 19:56:40.839528697 +0000 UTC m=+1601.839546781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/911ec323-a66d-4d26-92b0-92b84d418131-maas-api-tls") pod "maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" (UID: "911ec323-a66d-4d26-92b0-92b84d418131") : secret "maas-api-e2e-derive-da4a26de-serving-cert" not found Jul 05 19:56:40.340070 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.340035 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/911ec323-a66d-4d26-92b0-92b84d418131-openshift-service-ca\") pod \"maas-api-e2e-derive-da4a26de-6657cc8699-25k5m\" (UID: \"911ec323-a66d-4d26-92b0-92b84d418131\") " pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" Jul 05 19:56:40.350714 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.350681 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc7v2\" (UniqueName: \"kubernetes.io/projected/911ec323-a66d-4d26-92b0-92b84d418131-kube-api-access-qc7v2\") pod \"maas-api-e2e-derive-da4a26de-6657cc8699-25k5m\" (UID: \"911ec323-a66d-4d26-92b0-92b84d418131\") " pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" Jul 05 19:56:40.516003 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.515971 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:40.810871 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.810776 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:40.844610 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.844568 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/911ec323-a66d-4d26-92b0-92b84d418131-maas-api-tls\") pod \"maas-api-e2e-derive-da4a26de-6657cc8699-25k5m\" (UID: \"911ec323-a66d-4d26-92b0-92b84d418131\") " pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" Jul 05 19:56:40.847500 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.847455 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/911ec323-a66d-4d26-92b0-92b84d418131-maas-api-tls\") pod \"maas-api-e2e-derive-da4a26de-6657cc8699-25k5m\" (UID: \"911ec323-a66d-4d26-92b0-92b84d418131\") " pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" Jul 05 19:56:40.909413 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:40.909366 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:41.065136 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:41.065009 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" Jul 05 19:56:41.212125 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:41.212097 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m"] Jul 05 19:56:41.214495 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:56:41.214459 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911ec323_a66d_4d26_92b0_92b84d418131.slice/crio-b490d014ef7025bdb8385300c516f6bee5396affa478dbdeb2d5297e3098ea79 WatchSource:0}: Error finding container b490d014ef7025bdb8385300c516f6bee5396affa478dbdeb2d5297e3098ea79: Status 404 returned error can't find the container with id b490d014ef7025bdb8385300c516f6bee5396affa478dbdeb2d5297e3098ea79 Jul 05 19:56:41.352553 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:41.352462 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" event={"ID":"911ec323-a66d-4d26-92b0-92b84d418131","Type":"ContainerStarted","Data":"b490d014ef7025bdb8385300c516f6bee5396affa478dbdeb2d5297e3098ea79"} Jul 05 19:56:42.358448 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:42.358415 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" event={"ID":"911ec323-a66d-4d26-92b0-92b84d418131","Type":"ContainerStarted","Data":"a61451cd024389f3c072115b8b19c761f7e486cbc9c3895fea33219d32ff6e67"} Jul 05 19:56:42.358892 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:42.358468 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" Jul 05 19:56:42.376695 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:42.376647 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" podStartSLOduration=1.9312477540000001 podStartE2EDuration="2.37663206s" totalImagesPullingTime="445.384306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:56:42.375031765 +0000 UTC m=+1603.375049871" watchObservedRunningTime="2026-07-05 19:56:42.37663206 +0000 UTC m=+1603.376650166" Jul 05 19:56:43.051492 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:43.051448 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:44.505089 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:44.505010 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:44.697766 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:44.697726 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:44.760876 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:44.760787 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m"] Jul 05 19:56:44.761084 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:44.761031 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" podUID="911ec323-a66d-4d26-92b0-92b84d418131" containerName="maas-api" containerID="cri-o://a61451cd024389f3c072115b8b19c761f7e486cbc9c3895fea33219d32ff6e67" gracePeriod=30 Jul 05 19:56:44.786226 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:44.786194 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:44.836544 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:44.836505 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:45.015421 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.015353 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" Jul 05 19:56:45.186969 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.186926 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/911ec323-a66d-4d26-92b0-92b84d418131-openshift-service-ca\") pod \"911ec323-a66d-4d26-92b0-92b84d418131\" (UID: \"911ec323-a66d-4d26-92b0-92b84d418131\") " Jul 05 19:56:45.187210 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.187073 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/911ec323-a66d-4d26-92b0-92b84d418131-maas-api-tls\") pod \"911ec323-a66d-4d26-92b0-92b84d418131\" (UID: \"911ec323-a66d-4d26-92b0-92b84d418131\") " Jul 05 19:56:45.187210 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.187112 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qc7v2\" (UniqueName: \"kubernetes.io/projected/911ec323-a66d-4d26-92b0-92b84d418131-kube-api-access-qc7v2\") pod \"911ec323-a66d-4d26-92b0-92b84d418131\" (UID: \"911ec323-a66d-4d26-92b0-92b84d418131\") " Jul 05 19:56:45.187368 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.187341 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911ec323-a66d-4d26-92b0-92b84d418131-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "911ec323-a66d-4d26-92b0-92b84d418131" (UID: "911ec323-a66d-4d26-92b0-92b84d418131"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:56:45.187465 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.187443 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/911ec323-a66d-4d26-92b0-92b84d418131-openshift-service-ca\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:56:45.189241 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.189211 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911ec323-a66d-4d26-92b0-92b84d418131-kube-api-access-qc7v2" (OuterVolumeSpecName: "kube-api-access-qc7v2") pod "911ec323-a66d-4d26-92b0-92b84d418131" (UID: "911ec323-a66d-4d26-92b0-92b84d418131"). InnerVolumeSpecName "kube-api-access-qc7v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:56:45.189332 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.189299 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/911ec323-a66d-4d26-92b0-92b84d418131-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "911ec323-a66d-4d26-92b0-92b84d418131" (UID: "911ec323-a66d-4d26-92b0-92b84d418131"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:56:45.288603 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.288509 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/911ec323-a66d-4d26-92b0-92b84d418131-maas-api-tls\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:56:45.288603 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.288541 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc7v2\" (UniqueName: \"kubernetes.io/projected/911ec323-a66d-4d26-92b0-92b84d418131-kube-api-access-qc7v2\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:56:45.374843 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.374805 2573 generic.go:356] "Generic (PLEG): container finished" podID="911ec323-a66d-4d26-92b0-92b84d418131" containerID="a61451cd024389f3c072115b8b19c761f7e486cbc9c3895fea33219d32ff6e67" exitCode=0 Jul 05 19:56:45.375025 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.374880 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" Jul 05 19:56:45.375025 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.374897 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" event={"ID":"911ec323-a66d-4d26-92b0-92b84d418131","Type":"ContainerDied","Data":"a61451cd024389f3c072115b8b19c761f7e486cbc9c3895fea33219d32ff6e67"} Jul 05 19:56:45.375025 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.374977 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m" event={"ID":"911ec323-a66d-4d26-92b0-92b84d418131","Type":"ContainerDied","Data":"b490d014ef7025bdb8385300c516f6bee5396affa478dbdeb2d5297e3098ea79"} Jul 05 19:56:45.375025 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.375000 2573 scope.go:117] "RemoveContainer" containerID="a61451cd024389f3c072115b8b19c761f7e486cbc9c3895fea33219d32ff6e67" Jul 05 19:56:45.384596 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.384581 2573 scope.go:117] "RemoveContainer" containerID="a61451cd024389f3c072115b8b19c761f7e486cbc9c3895fea33219d32ff6e67" Jul 05 19:56:45.384875 ip-10-0-138-40 kubenswrapper[2573]: E0705 19:56:45.384855 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61451cd024389f3c072115b8b19c761f7e486cbc9c3895fea33219d32ff6e67\": container with ID starting with a61451cd024389f3c072115b8b19c761f7e486cbc9c3895fea33219d32ff6e67 not found: ID does not exist" containerID="a61451cd024389f3c072115b8b19c761f7e486cbc9c3895fea33219d32ff6e67" Jul 05 19:56:45.384927 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.384887 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61451cd024389f3c072115b8b19c761f7e486cbc9c3895fea33219d32ff6e67"} err="failed to get container status \"a61451cd024389f3c072115b8b19c761f7e486cbc9c3895fea33219d32ff6e67\": rpc error: code = NotFound desc = could not find container \"a61451cd024389f3c072115b8b19c761f7e486cbc9c3895fea33219d32ff6e67\": container with ID starting with a61451cd024389f3c072115b8b19c761f7e486cbc9c3895fea33219d32ff6e67 not found: ID does not exist" Jul 05 19:56:45.395925 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.395895 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m"] Jul 05 19:56:45.399668 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.399647 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-da4a26de-6657cc8699-25k5m"] Jul 05 19:56:45.469681 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:45.469649 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="911ec323-a66d-4d26-92b0-92b84d418131" path="/var/lib/kubelet/pods/911ec323-a66d-4d26-92b0-92b84d418131/volumes" Jul 05 19:56:46.846366 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:46.846324 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:50.618896 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:50.618818 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:50.681816 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:50.681780 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:51.083906 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:51.083872 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:51.180639 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:51.180602 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:51.282236 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:51.282199 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:52.617070 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:52.617012 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:58.483795 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:58.483763 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:58.541435 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:58.541406 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:58.611562 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:58.611529 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:58.710734 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:58.710699 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:58.813966 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:58.813919 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:56:59.157311 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:56:59.157220 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:57:25.110585 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:57:25.110547 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:57:30.706225 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:57:30.706191 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:57:30.813146 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:57:30.813106 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:57:30.908256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:57:30.908220 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:57:46.786289 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:57:46.786252 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:57:52.404528 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:57:52.404489 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:57:52.464186 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:57:52.464155 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:57:52.526662 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:57:52.526628 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:57:52.627610 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:57:52.627575 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:03.109830 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:03.109798 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:03.870393 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:03.870360 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:09.573193 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:09.573158 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:09.635002 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:09.634967 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:09.707474 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:09.707445 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:09.807408 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:09.807374 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:09.908421 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:09.908334 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:10.591938 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:10.591904 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:10.765256 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:10.765223 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:10.865489 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:10.865398 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:12.290235 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:12.290197 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:17.879273 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:17.879226 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:17.933361 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:17.933324 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:18.527069 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:18.527001 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:24.191597 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:24.191512 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:24.291979 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:24.291936 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:24.404977 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:24.404946 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:24.704359 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:24.704315 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:24.982249 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:24.982160 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:27.906946 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:27.906914 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:28.005180 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.005150 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:28.086022 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.085980 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z"] Jul 05 19:58:28.086644 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.086623 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="911ec323-a66d-4d26-92b0-92b84d418131" containerName="maas-api" Jul 05 19:58:28.086696 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.086650 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="911ec323-a66d-4d26-92b0-92b84d418131" containerName="maas-api" Jul 05 19:58:28.086788 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.086775 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="911ec323-a66d-4d26-92b0-92b84d418131" containerName="maas-api" Jul 05 19:58:28.090469 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.090448 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" Jul 05 19:58:28.093338 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.093305 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-a4a20bf3-serving-cert\"" Jul 05 19:58:28.100744 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.100712 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z"] Jul 05 19:58:28.122901 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.122862 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:28.141163 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.141124 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5s6f\" (UniqueName: \"kubernetes.io/projected/70a1530f-1b34-4b9d-aa4d-760178c1347e-kube-api-access-l5s6f\") pod \"maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z\" (UID: \"70a1530f-1b34-4b9d-aa4d-760178c1347e\") " pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" Jul 05 19:58:28.141320 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.141257 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a1530f-1b34-4b9d-aa4d-760178c1347e-openshift-service-ca\") pod \"maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z\" (UID: \"70a1530f-1b34-4b9d-aa4d-760178c1347e\") " pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" Jul 05 19:58:28.141320 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.141312 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a1530f-1b34-4b9d-aa4d-760178c1347e-maas-api-tls\") pod \"maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z\" (UID: \"70a1530f-1b34-4b9d-aa4d-760178c1347e\") " pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" Jul 05 19:58:28.242555 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.242451 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a1530f-1b34-4b9d-aa4d-760178c1347e-openshift-service-ca\") pod \"maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z\" (UID: \"70a1530f-1b34-4b9d-aa4d-760178c1347e\") " pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" Jul 05 19:58:28.242555 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.242516 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a1530f-1b34-4b9d-aa4d-760178c1347e-maas-api-tls\") pod \"maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z\" (UID: \"70a1530f-1b34-4b9d-aa4d-760178c1347e\") " pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" Jul 05 19:58:28.242786 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.242590 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5s6f\" (UniqueName: \"kubernetes.io/projected/70a1530f-1b34-4b9d-aa4d-760178c1347e-kube-api-access-l5s6f\") pod \"maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z\" (UID: \"70a1530f-1b34-4b9d-aa4d-760178c1347e\") " pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" Jul 05 19:58:28.243289 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.243266 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a1530f-1b34-4b9d-aa4d-760178c1347e-openshift-service-ca\") pod \"maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z\" (UID: \"70a1530f-1b34-4b9d-aa4d-760178c1347e\") " pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" Jul 05 19:58:28.245244 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.245220 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a1530f-1b34-4b9d-aa4d-760178c1347e-maas-api-tls\") pod \"maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z\" (UID: \"70a1530f-1b34-4b9d-aa4d-760178c1347e\") " pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" Jul 05 19:58:28.256766 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.256740 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5s6f\" (UniqueName: \"kubernetes.io/projected/70a1530f-1b34-4b9d-aa4d-760178c1347e-kube-api-access-l5s6f\") pod \"maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z\" (UID: \"70a1530f-1b34-4b9d-aa4d-760178c1347e\") " pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" Jul 05 19:58:28.278462 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.278403 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:28.402835 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.402797 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" Jul 05 19:58:28.548165 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.548140 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z"] Jul 05 19:58:28.551012 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:58:28.550991 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a1530f_1b34_4b9d_aa4d_760178c1347e.slice/crio-a83baab6bdcabfe12c3ad0843ef6b3553d7ec80acadfc3c33299f7b70950cb5f WatchSource:0}: Error finding container a83baab6bdcabfe12c3ad0843ef6b3553d7ec80acadfc3c33299f7b70950cb5f: Status 404 returned error can't find the container with id a83baab6bdcabfe12c3ad0843ef6b3553d7ec80acadfc3c33299f7b70950cb5f Jul 05 19:58:28.655701 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.655655 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:28.756061 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.756008 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:28.899301 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:28.899213 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" event={"ID":"70a1530f-1b34-4b9d-aa4d-760178c1347e","Type":"ContainerStarted","Data":"a83baab6bdcabfe12c3ad0843ef6b3553d7ec80acadfc3c33299f7b70950cb5f"} Jul 05 19:58:29.905888 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:29.905848 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" event={"ID":"70a1530f-1b34-4b9d-aa4d-760178c1347e","Type":"ContainerStarted","Data":"ffe8ded079d30009f80248faaf8e7aa7b278e8af5c83c9b8a29baeafd608d19d"} Jul 05 19:58:29.906407 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:29.905970 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" Jul 05 19:58:29.923139 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:29.923093 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" podStartSLOduration=1.502508926 podStartE2EDuration="1.923078852s" totalImagesPullingTime="420.569926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:58:29.921137416 +0000 UTC m=+1710.921155522" watchObservedRunningTime="2026-07-05 19:58:29.923078852 +0000 UTC m=+1710.923096957" Jul 05 19:58:32.379382 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:32.379328 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:33.009996 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:33.009964 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:33.110253 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:33.110218 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:33.445498 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:33.445415 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:35.915617 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:35.915587 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" Jul 05 19:58:39.207213 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:39.207182 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:39.308359 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:39.308322 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:39.589386 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:39.589350 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:39.807232 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:39.807197 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z"] Jul 05 19:58:39.807507 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:39.807452 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" podUID="70a1530f-1b34-4b9d-aa4d-760178c1347e" containerName="maas-api" containerID="cri-o://ffe8ded079d30009f80248faaf8e7aa7b278e8af5c83c9b8a29baeafd608d19d" gracePeriod=30 Jul 05 19:58:39.824692 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:39.824496 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:39.960283 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:39.960251 2573 generic.go:356] "Generic (PLEG): container finished" podID="70a1530f-1b34-4b9d-aa4d-760178c1347e" containerID="ffe8ded079d30009f80248faaf8e7aa7b278e8af5c83c9b8a29baeafd608d19d" exitCode=0 Jul 05 19:58:39.960495 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:39.960285 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" event={"ID":"70a1530f-1b34-4b9d-aa4d-760178c1347e","Type":"ContainerDied","Data":"ffe8ded079d30009f80248faaf8e7aa7b278e8af5c83c9b8a29baeafd608d19d"} Jul 05 19:58:39.964646 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:39.964621 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:40.066070 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:40.066027 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" Jul 05 19:58:40.152514 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:40.152475 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a1530f-1b34-4b9d-aa4d-760178c1347e-maas-api-tls\") pod \"70a1530f-1b34-4b9d-aa4d-760178c1347e\" (UID: \"70a1530f-1b34-4b9d-aa4d-760178c1347e\") " Jul 05 19:58:40.152687 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:40.152593 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5s6f\" (UniqueName: \"kubernetes.io/projected/70a1530f-1b34-4b9d-aa4d-760178c1347e-kube-api-access-l5s6f\") pod \"70a1530f-1b34-4b9d-aa4d-760178c1347e\" (UID: \"70a1530f-1b34-4b9d-aa4d-760178c1347e\") " Jul 05 19:58:40.152687 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:40.152611 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a1530f-1b34-4b9d-aa4d-760178c1347e-openshift-service-ca\") pod \"70a1530f-1b34-4b9d-aa4d-760178c1347e\" (UID: \"70a1530f-1b34-4b9d-aa4d-760178c1347e\") " Jul 05 19:58:40.153016 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:40.152993 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a1530f-1b34-4b9d-aa4d-760178c1347e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "70a1530f-1b34-4b9d-aa4d-760178c1347e" (UID: "70a1530f-1b34-4b9d-aa4d-760178c1347e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 19:58:40.154796 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:40.154770 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a1530f-1b34-4b9d-aa4d-760178c1347e-kube-api-access-l5s6f" (OuterVolumeSpecName: "kube-api-access-l5s6f") pod "70a1530f-1b34-4b9d-aa4d-760178c1347e" (UID: "70a1530f-1b34-4b9d-aa4d-760178c1347e"). InnerVolumeSpecName "kube-api-access-l5s6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 19:58:40.154796 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:40.154775 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70a1530f-1b34-4b9d-aa4d-760178c1347e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "70a1530f-1b34-4b9d-aa4d-760178c1347e" (UID: "70a1530f-1b34-4b9d-aa4d-760178c1347e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 19:58:40.253535 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:40.253503 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5s6f\" (UniqueName: \"kubernetes.io/projected/70a1530f-1b34-4b9d-aa4d-760178c1347e-kube-api-access-l5s6f\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:58:40.253535 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:40.253535 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a1530f-1b34-4b9d-aa4d-760178c1347e-openshift-service-ca\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:58:40.253929 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:40.253545 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70a1530f-1b34-4b9d-aa4d-760178c1347e-maas-api-tls\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 19:58:40.967206 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:40.967176 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" Jul 05 19:58:40.967383 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:40.967211 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z" event={"ID":"70a1530f-1b34-4b9d-aa4d-760178c1347e","Type":"ContainerDied","Data":"a83baab6bdcabfe12c3ad0843ef6b3553d7ec80acadfc3c33299f7b70950cb5f"} Jul 05 19:58:40.967383 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:40.967273 2573 scope.go:117] "RemoveContainer" containerID="ffe8ded079d30009f80248faaf8e7aa7b278e8af5c83c9b8a29baeafd608d19d" Jul 05 19:58:40.988593 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:40.988569 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z"] Jul 05 19:58:40.991964 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:40.991941 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-a4a20bf3-d789959f7-j2z6z"] Jul 05 19:58:41.469862 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:41.469827 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a1530f-1b34-4b9d-aa4d-760178c1347e" path="/var/lib/kubelet/pods/70a1530f-1b34-4b9d-aa4d-760178c1347e/volumes" Jul 05 19:58:45.569315 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:45.569283 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:45.626159 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:45.626127 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:45.810107 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:45.810077 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:46.087146 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:46.087112 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:46.179681 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:46.179645 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:46.380274 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:46.380166 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:51.924215 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:51.924117 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:51.975252 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:51.975206 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:52.046994 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:52.046955 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:52.146219 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:52.146172 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:52.277197 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:52.277167 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:57.843412 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:57.843375 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:58:57.892422 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:58:57.892388 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:04.532974 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:04.532935 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:04.820325 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:04.820203 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:05.051915 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:05.051868 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:08.075382 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.075346 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:08.138573 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.138539 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4"] Jul 05 19:59:08.139176 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.139153 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a1530f-1b34-4b9d-aa4d-760178c1347e" containerName="maas-api" Jul 05 19:59:08.139322 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.139236 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a1530f-1b34-4b9d-aa4d-760178c1347e" containerName="maas-api" Jul 05 19:59:08.139412 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.139393 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a1530f-1b34-4b9d-aa4d-760178c1347e" containerName="maas-api" Jul 05 19:59:08.142648 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.142616 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" Jul 05 19:59:08.146963 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.146939 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-5592633f-serving-cert\"" Jul 05 19:59:08.154601 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.154566 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4"] Jul 05 19:59:08.155289 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.155237 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:08.220274 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.220242 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:08.325220 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.325171 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-openshift-service-ca\") pod \"maas-api-e2e-mt-5592633f-69b855f96f-ggmf4\" (UID: \"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f\") " pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" Jul 05 19:59:08.325441 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.325237 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-maas-api-tls\") pod \"maas-api-e2e-mt-5592633f-69b855f96f-ggmf4\" (UID: \"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f\") " pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" Jul 05 19:59:08.325441 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.325333 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbw4v\" (UniqueName: \"kubernetes.io/projected/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-kube-api-access-jbw4v\") pod \"maas-api-e2e-mt-5592633f-69b855f96f-ggmf4\" (UID: \"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f\") " pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" Jul 05 19:59:08.327272 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.327254 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:08.426377 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.426341 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-openshift-service-ca\") pod \"maas-api-e2e-mt-5592633f-69b855f96f-ggmf4\" (UID: \"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f\") " pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" Jul 05 19:59:08.426564 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.426399 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-maas-api-tls\") pod \"maas-api-e2e-mt-5592633f-69b855f96f-ggmf4\" (UID: \"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f\") " pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" Jul 05 19:59:08.426564 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.426463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbw4v\" (UniqueName: \"kubernetes.io/projected/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-kube-api-access-jbw4v\") pod \"maas-api-e2e-mt-5592633f-69b855f96f-ggmf4\" (UID: \"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f\") " pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" Jul 05 19:59:08.427093 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.427035 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-openshift-service-ca\") pod \"maas-api-e2e-mt-5592633f-69b855f96f-ggmf4\" (UID: \"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f\") " pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" Jul 05 19:59:08.429084 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.429064 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-maas-api-tls\") pod \"maas-api-e2e-mt-5592633f-69b855f96f-ggmf4\" (UID: \"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f\") " pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" Jul 05 19:59:08.434759 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.434739 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbw4v\" (UniqueName: \"kubernetes.io/projected/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-kube-api-access-jbw4v\") pod \"maas-api-e2e-mt-5592633f-69b855f96f-ggmf4\" (UID: \"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f\") " pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" Jul 05 19:59:08.454705 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.454670 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" Jul 05 19:59:08.600900 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.600872 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4"] Jul 05 19:59:08.603339 ip-10-0-138-40 kubenswrapper[2573]: W0705 19:59:08.603306 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3ef97c_4ad1_4ce3_a3bc_834387c1ed1f.slice/crio-2b6512e5b1fee8162e9de77e3e94478588d3fbdc9d57dddbeeafd3ae7d950ea0 WatchSource:0}: Error finding container 2b6512e5b1fee8162e9de77e3e94478588d3fbdc9d57dddbeeafd3ae7d950ea0: Status 404 returned error can't find the container with id 2b6512e5b1fee8162e9de77e3e94478588d3fbdc9d57dddbeeafd3ae7d950ea0 Jul 05 19:59:08.616167 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.616141 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:08.727445 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:08.727415 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:09.109216 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:09.109180 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" event={"ID":"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f","Type":"ContainerStarted","Data":"fe7f798c3064ea95cdc916be146c69ee69f7047e8b010cca2f6232b2060e752d"} Jul 05 19:59:09.109689 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:09.109246 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" Jul 05 19:59:09.109689 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:09.109269 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" event={"ID":"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f","Type":"ContainerStarted","Data":"2b6512e5b1fee8162e9de77e3e94478588d3fbdc9d57dddbeeafd3ae7d950ea0"} Jul 05 19:59:09.126906 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:09.126837 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" podStartSLOduration=0.732995324 podStartE2EDuration="1.126818149s" totalImagesPullingTime="393.822825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 19:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 19:59:09.12553149 +0000 UTC m=+1750.125549600" watchObservedRunningTime="2026-07-05 19:59:09.126818149 +0000 UTC m=+1750.126836262" Jul 05 19:59:12.562023 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:12.561991 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:15.122935 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:15.122908 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" Jul 05 19:59:19.292731 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:19.292696 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:19.393235 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:19.393204 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:22.269634 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:22.269593 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:35.923160 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:35.923127 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:36.021820 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:36.021774 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 19:59:59.589775 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:59.589737 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 19:59:59.594139 ip-10-0-138-40 kubenswrapper[2573]: I0705 19:59:59.594112 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 20:00:00.129510 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.129475 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t"] Jul 05 20:00:00.133166 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.133144 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29721360-n9kkh"] Jul 05 20:00:00.133298 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.133257 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t" Jul 05 20:00:00.136661 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.136641 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29721360-n9kkh" Jul 05 20:00:00.139023 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.138998 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t"] Jul 05 20:00:00.142319 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.142019 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29721360-n9kkh"] Jul 05 20:00:00.185171 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.185132 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvdnk\" (UniqueName: \"kubernetes.io/projected/aca77cbd-0ba3-4984-ab57-9d3ceef15317-kube-api-access-fvdnk\") pod \"maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t\" (UID: \"aca77cbd-0ba3-4984-ab57-9d3ceef15317\") " pod="opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t" Jul 05 20:00:00.185320 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.185176 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb9gz\" (UniqueName: \"kubernetes.io/projected/190e4082-b68d-4a36-b36d-fae0208f214a-kube-api-access-pb9gz\") pod \"maas-api-key-cleanup-29721360-n9kkh\" (UID: \"190e4082-b68d-4a36-b36d-fae0208f214a\") " pod="opendatahub/maas-api-key-cleanup-29721360-n9kkh" Jul 05 20:00:00.269369 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.269343 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:00.286515 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.286489 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb9gz\" (UniqueName: \"kubernetes.io/projected/190e4082-b68d-4a36-b36d-fae0208f214a-kube-api-access-pb9gz\") pod \"maas-api-key-cleanup-29721360-n9kkh\" (UID: \"190e4082-b68d-4a36-b36d-fae0208f214a\") " pod="opendatahub/maas-api-key-cleanup-29721360-n9kkh" Jul 05 20:00:00.286610 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.286562 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvdnk\" (UniqueName: \"kubernetes.io/projected/aca77cbd-0ba3-4984-ab57-9d3ceef15317-kube-api-access-fvdnk\") pod \"maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t\" (UID: \"aca77cbd-0ba3-4984-ab57-9d3ceef15317\") " pod="opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t" Jul 05 20:00:00.294869 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.294845 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb9gz\" (UniqueName: \"kubernetes.io/projected/190e4082-b68d-4a36-b36d-fae0208f214a-kube-api-access-pb9gz\") pod \"maas-api-key-cleanup-29721360-n9kkh\" (UID: \"190e4082-b68d-4a36-b36d-fae0208f214a\") " pod="opendatahub/maas-api-key-cleanup-29721360-n9kkh" Jul 05 20:00:00.294869 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.294862 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvdnk\" (UniqueName: \"kubernetes.io/projected/aca77cbd-0ba3-4984-ab57-9d3ceef15317-kube-api-access-fvdnk\") pod \"maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t\" (UID: \"aca77cbd-0ba3-4984-ab57-9d3ceef15317\") " pod="opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t" Jul 05 20:00:00.447903 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.447817 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t" Jul 05 20:00:00.455867 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.455846 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29721360-n9kkh" Jul 05 20:00:00.587841 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.587804 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t"] Jul 05 20:00:00.592182 ip-10-0-138-40 kubenswrapper[2573]: W0705 20:00:00.592144 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca77cbd_0ba3_4984_ab57_9d3ceef15317.slice/crio-fedc0b3344306ca54b95c031d6ddb1029295b0a0fda24ad7c56a6936f3c5f0e7 WatchSource:0}: Error finding container fedc0b3344306ca54b95c031d6ddb1029295b0a0fda24ad7c56a6936f3c5f0e7: Status 404 returned error can't find the container with id fedc0b3344306ca54b95c031d6ddb1029295b0a0fda24ad7c56a6936f3c5f0e7 Jul 05 20:00:00.613865 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.613842 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29721360-n9kkh"] Jul 05 20:00:00.615726 ip-10-0-138-40 kubenswrapper[2573]: W0705 20:00:00.615703 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190e4082_b68d_4a36_b36d_fae0208f214a.slice/crio-b8962d7c6e595ce60f079ffe1d2c593938718980b23f73feb7e496f0a5228448 WatchSource:0}: Error finding container b8962d7c6e595ce60f079ffe1d2c593938718980b23f73feb7e496f0a5228448: Status 404 returned error can't find the container with id b8962d7c6e595ce60f079ffe1d2c593938718980b23f73feb7e496f0a5228448 Jul 05 20:00:00.772031 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.772005 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:00.872660 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.872637 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:00.942422 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.940454 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4"] Jul 05 20:00:00.942422 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.940735 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" podUID="df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f" containerName="maas-api" containerID="cri-o://fe7f798c3064ea95cdc916be146c69ee69f7047e8b010cca2f6232b2060e752d" gracePeriod=30 Jul 05 20:00:00.971861 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.971837 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t"] Jul 05 20:00:00.978855 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:00.978823 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:01.174573 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.174550 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" Jul 05 20:00:01.195863 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.195840 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-openshift-service-ca\") pod \"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f\" (UID: \"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f\") " Jul 05 20:00:01.195999 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.195985 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-maas-api-tls\") pod \"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f\" (UID: \"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f\") " Jul 05 20:00:01.196092 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.196025 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbw4v\" (UniqueName: \"kubernetes.io/projected/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-kube-api-access-jbw4v\") pod \"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f\" (UID: \"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f\") " Jul 05 20:00:01.196246 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.196221 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f" (UID: "df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 20:00:01.196327 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.196312 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-openshift-service-ca\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:00:01.198133 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.198110 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f" (UID: "df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:00:01.198291 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.198269 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-kube-api-access-jbw4v" (OuterVolumeSpecName: "kube-api-access-jbw4v") pod "df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f" (UID: "df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f"). InnerVolumeSpecName "kube-api-access-jbw4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:00:01.297672 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.297641 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-maas-api-tls\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:00:01.297672 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.297670 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbw4v\" (UniqueName: \"kubernetes.io/projected/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f-kube-api-access-jbw4v\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:00:01.371583 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.371548 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29721360-n9kkh" event={"ID":"190e4082-b68d-4a36-b36d-fae0208f214a","Type":"ContainerStarted","Data":"2375a346cecf548950936bd6fc81255f02d460765be613570393d75ece249918"} Jul 05 20:00:01.371730 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.371606 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29721360-n9kkh" event={"ID":"190e4082-b68d-4a36-b36d-fae0208f214a","Type":"ContainerStarted","Data":"b8962d7c6e595ce60f079ffe1d2c593938718980b23f73feb7e496f0a5228448"} Jul 05 20:00:01.373062 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.373015 2573 generic.go:356] "Generic (PLEG): container finished" podID="aca77cbd-0ba3-4984-ab57-9d3ceef15317" containerID="ab81efc272e1f34b70ff3bd83a65e6240a284aaec1ad2c7d0f5996c3651b356e" exitCode=6 Jul 05 20:00:01.373149 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.373119 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t" event={"ID":"aca77cbd-0ba3-4984-ab57-9d3ceef15317","Type":"ContainerDied","Data":"ab81efc272e1f34b70ff3bd83a65e6240a284aaec1ad2c7d0f5996c3651b356e"} Jul 05 20:00:01.373235 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.373148 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t" event={"ID":"aca77cbd-0ba3-4984-ab57-9d3ceef15317","Type":"ContainerStarted","Data":"fedc0b3344306ca54b95c031d6ddb1029295b0a0fda24ad7c56a6936f3c5f0e7"} Jul 05 20:00:01.373235 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.373190 2573 scope.go:117] "RemoveContainer" containerID="ab81efc272e1f34b70ff3bd83a65e6240a284aaec1ad2c7d0f5996c3651b356e" Jul 05 20:00:01.374360 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.374340 2573 generic.go:356] "Generic (PLEG): container finished" podID="df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f" containerID="fe7f798c3064ea95cdc916be146c69ee69f7047e8b010cca2f6232b2060e752d" exitCode=0 Jul 05 20:00:01.374455 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.374384 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" Jul 05 20:00:01.374455 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.374399 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" event={"ID":"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f","Type":"ContainerDied","Data":"fe7f798c3064ea95cdc916be146c69ee69f7047e8b010cca2f6232b2060e752d"} Jul 05 20:00:01.374455 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.374427 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4" event={"ID":"df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f","Type":"ContainerDied","Data":"2b6512e5b1fee8162e9de77e3e94478588d3fbdc9d57dddbeeafd3ae7d950ea0"} Jul 05 20:00:01.374455 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.374449 2573 scope.go:117] "RemoveContainer" containerID="fe7f798c3064ea95cdc916be146c69ee69f7047e8b010cca2f6232b2060e752d" Jul 05 20:00:01.389504 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.389341 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29721360-n9kkh" podStartSLOduration=1.389323413 podStartE2EDuration="1.389323413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:00:01.388723335 +0000 UTC m=+1802.388741440" watchObservedRunningTime="2026-07-05 20:00:01.389323413 +0000 UTC m=+1802.389341519" Jul 05 20:00:01.402958 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.402939 2573 scope.go:117] "RemoveContainer" containerID="fe7f798c3064ea95cdc916be146c69ee69f7047e8b010cca2f6232b2060e752d" Jul 05 20:00:01.403318 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:00:01.403295 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7f798c3064ea95cdc916be146c69ee69f7047e8b010cca2f6232b2060e752d\": container with ID starting with fe7f798c3064ea95cdc916be146c69ee69f7047e8b010cca2f6232b2060e752d not found: ID does not exist" containerID="fe7f798c3064ea95cdc916be146c69ee69f7047e8b010cca2f6232b2060e752d" Jul 05 20:00:01.403409 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.403323 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7f798c3064ea95cdc916be146c69ee69f7047e8b010cca2f6232b2060e752d"} err="failed to get container status \"fe7f798c3064ea95cdc916be146c69ee69f7047e8b010cca2f6232b2060e752d\": rpc error: code = NotFound desc = could not find container \"fe7f798c3064ea95cdc916be146c69ee69f7047e8b010cca2f6232b2060e752d\": container with ID starting with fe7f798c3064ea95cdc916be146c69ee69f7047e8b010cca2f6232b2060e752d not found: ID does not exist" Jul 05 20:00:01.426399 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.426376 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4"] Jul 05 20:00:01.430804 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.430783 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-5592633f-69b855f96f-ggmf4"] Jul 05 20:00:01.469716 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:01.469690 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f" path="/var/lib/kubelet/pods/df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f/volumes" Jul 05 20:00:02.380741 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:02.380704 2573 generic.go:356] "Generic (PLEG): container finished" podID="aca77cbd-0ba3-4984-ab57-9d3ceef15317" containerID="ee43f74e8f266b3e0c6529bcffd87cd2370a0a1bc19ec7c085fcd1935efadf0c" exitCode=6 Jul 05 20:00:02.381171 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:02.380789 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t" event={"ID":"aca77cbd-0ba3-4984-ab57-9d3ceef15317","Type":"ContainerDied","Data":"ee43f74e8f266b3e0c6529bcffd87cd2370a0a1bc19ec7c085fcd1935efadf0c"} Jul 05 20:00:02.381171 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:02.380878 2573 scope.go:117] "RemoveContainer" containerID="ab81efc272e1f34b70ff3bd83a65e6240a284aaec1ad2c7d0f5996c3651b356e" Jul 05 20:00:02.383234 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:02.383213 2573 generic.go:356] "Generic (PLEG): container finished" podID="190e4082-b68d-4a36-b36d-fae0208f214a" containerID="2375a346cecf548950936bd6fc81255f02d460765be613570393d75ece249918" exitCode=0 Jul 05 20:00:02.383340 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:02.383292 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29721360-n9kkh" event={"ID":"190e4082-b68d-4a36-b36d-fae0208f214a","Type":"ContainerDied","Data":"2375a346cecf548950936bd6fc81255f02d460765be613570393d75ece249918"} Jul 05 20:00:02.514543 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:02.514520 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t" Jul 05 20:00:02.610164 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:02.610124 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvdnk\" (UniqueName: \"kubernetes.io/projected/aca77cbd-0ba3-4984-ab57-9d3ceef15317-kube-api-access-fvdnk\") pod \"aca77cbd-0ba3-4984-ab57-9d3ceef15317\" (UID: \"aca77cbd-0ba3-4984-ab57-9d3ceef15317\") " Jul 05 20:00:02.612265 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:02.612242 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca77cbd-0ba3-4984-ab57-9d3ceef15317-kube-api-access-fvdnk" (OuterVolumeSpecName: "kube-api-access-fvdnk") pod "aca77cbd-0ba3-4984-ab57-9d3ceef15317" (UID: "aca77cbd-0ba3-4984-ab57-9d3ceef15317"). InnerVolumeSpecName "kube-api-access-fvdnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:00:02.711331 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:02.711255 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvdnk\" (UniqueName: \"kubernetes.io/projected/aca77cbd-0ba3-4984-ab57-9d3ceef15317-kube-api-access-fvdnk\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:00:03.390058 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:03.390006 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t" event={"ID":"aca77cbd-0ba3-4984-ab57-9d3ceef15317","Type":"ContainerDied","Data":"fedc0b3344306ca54b95c031d6ddb1029295b0a0fda24ad7c56a6936f3c5f0e7"} Jul 05 20:00:03.390588 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:03.390085 2573 scope.go:117] "RemoveContainer" containerID="ee43f74e8f266b3e0c6529bcffd87cd2370a0a1bc19ec7c085fcd1935efadf0c" Jul 05 20:00:03.390588 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:03.390020 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t" Jul 05 20:00:03.414253 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:03.414230 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t"] Jul 05 20:00:03.420158 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:03.420136 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-5592633f-29721360-6tf9t"] Jul 05 20:00:03.471365 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:03.471340 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aca77cbd-0ba3-4984-ab57-9d3ceef15317" path="/var/lib/kubelet/pods/aca77cbd-0ba3-4984-ab57-9d3ceef15317/volumes" Jul 05 20:00:03.521775 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:03.521750 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29721360-n9kkh" Jul 05 20:00:03.621289 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:03.621259 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb9gz\" (UniqueName: \"kubernetes.io/projected/190e4082-b68d-4a36-b36d-fae0208f214a-kube-api-access-pb9gz\") pod \"190e4082-b68d-4a36-b36d-fae0208f214a\" (UID: \"190e4082-b68d-4a36-b36d-fae0208f214a\") " Jul 05 20:00:03.623397 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:03.623369 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190e4082-b68d-4a36-b36d-fae0208f214a-kube-api-access-pb9gz" (OuterVolumeSpecName: "kube-api-access-pb9gz") pod "190e4082-b68d-4a36-b36d-fae0208f214a" (UID: "190e4082-b68d-4a36-b36d-fae0208f214a"). InnerVolumeSpecName "kube-api-access-pb9gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:00:03.722565 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:03.722500 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb9gz\" (UniqueName: \"kubernetes.io/projected/190e4082-b68d-4a36-b36d-fae0208f214a-kube-api-access-pb9gz\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:00:04.400461 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:04.400431 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29721360-n9kkh" Jul 05 20:00:04.400814 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:04.400433 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29721360-n9kkh" event={"ID":"190e4082-b68d-4a36-b36d-fae0208f214a","Type":"ContainerDied","Data":"b8962d7c6e595ce60f079ffe1d2c593938718980b23f73feb7e496f0a5228448"} Jul 05 20:00:04.400814 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:04.400564 2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8962d7c6e595ce60f079ffe1d2c593938718980b23f73feb7e496f0a5228448" Jul 05 20:00:06.262500 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:06.262466 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:11.346658 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:11.346622 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:47.603970 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:47.603938 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:48.154800 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:48.154768 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:53.599331 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:53.599285 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:53.662180 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:53.662149 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:53.954515 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:53.954433 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:54.113866 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:54.113832 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:55.118318 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:55.118289 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:55.217725 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:55.217697 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:55.320224 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:55.320195 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:56.022215 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:56.022182 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:56.113770 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:56.113736 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:56.213826 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:56.213798 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:00:57.616284 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:00:57.616254 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:03.293488 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:03.293448 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:03.394707 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:03.394678 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:03.950739 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:03.950710 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:09.565032 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:09.564997 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:09.664706 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:09.664665 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:09.764764 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:09.764733 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:09.900740 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:09.900652 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:20.434581 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:20.434464 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:20.797730 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:20.797697 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:41.717001 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:41.716968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:47.380766 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:47.380723 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:47.480104 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:47.480068 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:47.729195 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:47.729116 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:48.063128 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:48.063093 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:48.125194 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:48.125153 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:50.800208 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.800173 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:50.899702 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.899669 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:50.984563 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.984523 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z"] Jul 05 20:01:50.985023 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.985003 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aca77cbd-0ba3-4984-ab57-9d3ceef15317" containerName="cleanup" Jul 05 20:01:50.985146 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.985026 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aca77cbd-0ba3-4984-ab57-9d3ceef15317" containerName="cleanup" Jul 05 20:01:50.985146 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.985073 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="190e4082-b68d-4a36-b36d-fae0208f214a" containerName="cleanup" Jul 05 20:01:50.985146 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.985083 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="190e4082-b68d-4a36-b36d-fae0208f214a" containerName="cleanup" Jul 05 20:01:50.985146 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.985097 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f" containerName="maas-api" Jul 05 20:01:50.985146 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.985106 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f" containerName="maas-api" Jul 05 20:01:50.985418 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.985237 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="df3ef97c-4ad1-4ce3-a3bc-834387c1ed1f" containerName="maas-api" Jul 05 20:01:50.985418 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.985250 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="190e4082-b68d-4a36-b36d-fae0208f214a" containerName="cleanup" Jul 05 20:01:50.985418 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.985261 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aca77cbd-0ba3-4984-ab57-9d3ceef15317" containerName="cleanup" Jul 05 20:01:50.985418 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.985270 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aca77cbd-0ba3-4984-ab57-9d3ceef15317" containerName="cleanup" Jul 05 20:01:50.988602 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.988578 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" Jul 05 20:01:50.991367 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.991343 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-6354d0-serving-cert\"" Jul 05 20:01:50.998807 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:50.998783 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z"] Jul 05 20:01:51.034292 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.034255 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:51.097736 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.097635 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfcpj\" (UniqueName: \"kubernetes.io/projected/d6448e97-aa4b-4490-9039-9c377902ab55-kube-api-access-kfcpj\") pod \"maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z\" (UID: \"d6448e97-aa4b-4490-9039-9c377902ab55\") " pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" Jul 05 20:01:51.097736 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.097699 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6448e97-aa4b-4490-9039-9c377902ab55-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z\" (UID: \"d6448e97-aa4b-4490-9039-9c377902ab55\") " pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" Jul 05 20:01:51.097972 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.097838 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6448e97-aa4b-4490-9039-9c377902ab55-maas-api-tls\") pod \"maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z\" (UID: \"d6448e97-aa4b-4490-9039-9c377902ab55\") " pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" Jul 05 20:01:51.198691 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.198651 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6448e97-aa4b-4490-9039-9c377902ab55-maas-api-tls\") pod \"maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z\" (UID: \"d6448e97-aa4b-4490-9039-9c377902ab55\") " pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" Jul 05 20:01:51.198873 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.198712 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfcpj\" (UniqueName: \"kubernetes.io/projected/d6448e97-aa4b-4490-9039-9c377902ab55-kube-api-access-kfcpj\") pod \"maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z\" (UID: \"d6448e97-aa4b-4490-9039-9c377902ab55\") " pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" Jul 05 20:01:51.198873 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.198749 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6448e97-aa4b-4490-9039-9c377902ab55-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z\" (UID: \"d6448e97-aa4b-4490-9039-9c377902ab55\") " pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" Jul 05 20:01:51.199425 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.199406 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6448e97-aa4b-4490-9039-9c377902ab55-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z\" (UID: \"d6448e97-aa4b-4490-9039-9c377902ab55\") " pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" Jul 05 20:01:51.201250 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.201231 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6448e97-aa4b-4490-9039-9c377902ab55-maas-api-tls\") pod \"maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z\" (UID: \"d6448e97-aa4b-4490-9039-9c377902ab55\") " pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" Jul 05 20:01:51.206878 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.206836 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:51.207568 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.207551 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfcpj\" (UniqueName: \"kubernetes.io/projected/d6448e97-aa4b-4490-9039-9c377902ab55-kube-api-access-kfcpj\") pod \"maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z\" (UID: \"d6448e97-aa4b-4490-9039-9c377902ab55\") " pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" Jul 05 20:01:51.303382 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.303344 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" Jul 05 20:01:51.434390 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.434300 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:51.449354 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.449319 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z"] Jul 05 20:01:51.453174 ip-10-0-138-40 kubenswrapper[2573]: W0705 20:01:51.453141 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6448e97_aa4b_4490_9039_9c377902ab55.slice/crio-a49e0af57b80b322111a5e6688a434ea4a2e5a9997538253d18a249fd1aaabb3 WatchSource:0}: Error finding container a49e0af57b80b322111a5e6688a434ea4a2e5a9997538253d18a249fd1aaabb3: Status 404 returned error can't find the container with id a49e0af57b80b322111a5e6688a434ea4a2e5a9997538253d18a249fd1aaabb3 Jul 05 20:01:51.454535 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.454514 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 05 20:01:51.535916 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.535880 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:51.928017 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.927984 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" event={"ID":"d6448e97-aa4b-4490-9039-9c377902ab55","Type":"ContainerStarted","Data":"e4918f9fa2ed14bc48e4cef46336860bd851f4e0a05812c1a8742db88c2497e5"} Jul 05 20:01:51.928430 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.928066 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" event={"ID":"d6448e97-aa4b-4490-9039-9c377902ab55","Type":"ContainerStarted","Data":"a49e0af57b80b322111a5e6688a434ea4a2e5a9997538253d18a249fd1aaabb3"} Jul 05 20:01:51.928430 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.928210 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" Jul 05 20:01:51.945977 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:51.945920 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" podStartSLOduration=1.5691856450000001 podStartE2EDuration="1.945899218s" totalImagesPullingTime="376.713573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:01:51.94376865 +0000 UTC m=+1912.943786756" watchObservedRunningTime="2026-07-05 20:01:51.945899218 +0000 UTC m=+1912.945917324" Jul 05 20:01:55.298108 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:55.298068 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:55.625210 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:55.625122 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:55.900810 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:55.900778 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:55.960910 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:55.960875 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:57.939408 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:57.939382 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" Jul 05 20:01:58.842835 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:58.842798 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:58.904407 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:58.904372 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l"] Jul 05 20:01:58.904789 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:58.904765 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aca77cbd-0ba3-4984-ab57-9d3ceef15317" containerName="cleanup" Jul 05 20:01:58.904789 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:58.904780 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aca77cbd-0ba3-4984-ab57-9d3ceef15317" containerName="cleanup" Jul 05 20:01:58.908316 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:58.908296 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:01:58.911186 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:58.911165 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-82d890-serving-cert\"" Jul 05 20:01:58.917785 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:58.917762 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l"] Jul 05 20:01:58.919953 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:58.919933 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:58.978317 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:58.978288 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw6r2\" (UniqueName: \"kubernetes.io/projected/0e20713c-6303-4028-9183-349302235ab9-kube-api-access-dw6r2\") pod \"maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l\" (UID: \"0e20713c-6303-4028-9183-349302235ab9\") " pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:01:58.978727 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:58.978339 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e20713c-6303-4028-9183-349302235ab9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l\" (UID: \"0e20713c-6303-4028-9183-349302235ab9\") " pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:01:58.978727 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:58.978422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e20713c-6303-4028-9183-349302235ab9-maas-api-tls\") pod \"maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l\" (UID: \"0e20713c-6303-4028-9183-349302235ab9\") " pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:01:58.979963 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:58.979878 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:59.079287 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:59.079248 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e20713c-6303-4028-9183-349302235ab9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l\" (UID: \"0e20713c-6303-4028-9183-349302235ab9\") " pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:01:59.079472 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:59.079326 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e20713c-6303-4028-9183-349302235ab9-maas-api-tls\") pod \"maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l\" (UID: \"0e20713c-6303-4028-9183-349302235ab9\") " pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:01:59.079472 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:59.079364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6r2\" (UniqueName: \"kubernetes.io/projected/0e20713c-6303-4028-9183-349302235ab9-kube-api-access-dw6r2\") pod \"maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l\" (UID: \"0e20713c-6303-4028-9183-349302235ab9\") " pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:01:59.079552 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:01:59.079474 2573 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-82d890-serving-cert: secret "maas-api-e2e-inf-b-82d890-serving-cert" not found Jul 05 20:01:59.079587 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:01:59.079568 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e20713c-6303-4028-9183-349302235ab9-maas-api-tls podName:0e20713c-6303-4028-9183-349302235ab9 nodeName:}" failed. No retries permitted until 2026-07-05 20:01:59.579551664 +0000 UTC m=+1920.579569748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e20713c-6303-4028-9183-349302235ab9-maas-api-tls") pod "maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" (UID: "0e20713c-6303-4028-9183-349302235ab9") : secret "maas-api-e2e-inf-b-82d890-serving-cert" not found Jul 05 20:01:59.079970 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:59.079950 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e20713c-6303-4028-9183-349302235ab9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l\" (UID: \"0e20713c-6303-4028-9183-349302235ab9\") " pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:01:59.090264 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:59.090240 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6r2\" (UniqueName: \"kubernetes.io/projected/0e20713c-6303-4028-9183-349302235ab9-kube-api-access-dw6r2\") pod \"maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l\" (UID: \"0e20713c-6303-4028-9183-349302235ab9\") " pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:01:59.287391 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:59.287357 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:59.459600 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:59.459567 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:59.559450 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:59.559348 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:01:59.584959 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:59.584926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e20713c-6303-4028-9183-349302235ab9-maas-api-tls\") pod \"maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l\" (UID: \"0e20713c-6303-4028-9183-349302235ab9\") " pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:01:59.587652 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:59.587618 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e20713c-6303-4028-9183-349302235ab9-maas-api-tls\") pod \"maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l\" (UID: \"0e20713c-6303-4028-9183-349302235ab9\") " pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:01:59.821530 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:59.821449 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:01:59.966626 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:01:59.966601 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l"] Jul 05 20:01:59.969119 ip-10-0-138-40 kubenswrapper[2573]: W0705 20:01:59.969085 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e20713c_6303_4028_9183_349302235ab9.slice/crio-f39bc028aff6bde71af1ffb9f0f1fc3dc2a5db3c60feceb5102650cf30ef7b5c WatchSource:0}: Error finding container f39bc028aff6bde71af1ffb9f0f1fc3dc2a5db3c60feceb5102650cf30ef7b5c: Status 404 returned error can't find the container with id f39bc028aff6bde71af1ffb9f0f1fc3dc2a5db3c60feceb5102650cf30ef7b5c Jul 05 20:02:00.979161 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:02:00.979117 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" event={"ID":"0e20713c-6303-4028-9183-349302235ab9","Type":"ContainerStarted","Data":"b7efdd9596f7652d45971df104392f64b612b8d73db0e7218268ae0f96f224d7"} Jul 05 20:02:00.979598 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:02:00.979177 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" event={"ID":"0e20713c-6303-4028-9183-349302235ab9","Type":"ContainerStarted","Data":"f39bc028aff6bde71af1ffb9f0f1fc3dc2a5db3c60feceb5102650cf30ef7b5c"} Jul 05 20:02:00.979598 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:02:00.979277 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:02:00.995704 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:02:00.995651 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" podStartSLOduration=2.510338091 podStartE2EDuration="2.995635s" totalImagesPullingTime="485.296909ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:02:00.994440533 +0000 UTC m=+1921.994458651" watchObservedRunningTime="2026-07-05 20:02:00.995635 +0000 UTC m=+1921.995653147" Jul 05 20:02:03.340747 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:02:03.340701 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:02:06.989136 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:02:06.989111 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:02:12.164810 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:02:12.164772 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:02:12.261945 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:02:12.261908 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:02:18.194801 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:02:18.194716 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:02:20.873458 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:02:20.873424 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:02:20.973715 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:02:20.973681 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:02:23.664782 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:02:23.664752 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:02:47.405026 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:02:47.404985 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:02:51.931273 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:02:51.931196 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:03:11.950090 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:11.950027 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:03:12.846157 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:12.846122 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:03:12.992512 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:12.992480 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:03:13.368334 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:13.368288 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:03:13.469520 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:13.469491 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:03:13.562624 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:13.562592 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z"] Jul 05 20:03:13.562834 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:13.562808 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" podUID="d6448e97-aa4b-4490-9039-9c377902ab55" containerName="maas-api" containerID="cri-o://e4918f9fa2ed14bc48e4cef46336860bd851f4e0a05812c1a8742db88c2497e5" gracePeriod=30 Jul 05 20:03:13.584073 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:13.584027 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:03:13.815352 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:13.815330 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" Jul 05 20:03:13.826219 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:13.826181 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfcpj\" (UniqueName: \"kubernetes.io/projected/d6448e97-aa4b-4490-9039-9c377902ab55-kube-api-access-kfcpj\") pod \"d6448e97-aa4b-4490-9039-9c377902ab55\" (UID: \"d6448e97-aa4b-4490-9039-9c377902ab55\") " Jul 05 20:03:13.826380 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:13.826301 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6448e97-aa4b-4490-9039-9c377902ab55-openshift-service-ca\") pod \"d6448e97-aa4b-4490-9039-9c377902ab55\" (UID: \"d6448e97-aa4b-4490-9039-9c377902ab55\") " Jul 05 20:03:13.826380 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:13.826334 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6448e97-aa4b-4490-9039-9c377902ab55-maas-api-tls\") pod \"d6448e97-aa4b-4490-9039-9c377902ab55\" (UID: \"d6448e97-aa4b-4490-9039-9c377902ab55\") " Jul 05 20:03:13.826739 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:13.826705 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6448e97-aa4b-4490-9039-9c377902ab55-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d6448e97-aa4b-4490-9039-9c377902ab55" (UID: "d6448e97-aa4b-4490-9039-9c377902ab55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 20:03:13.828483 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:13.828451 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6448e97-aa4b-4490-9039-9c377902ab55-kube-api-access-kfcpj" (OuterVolumeSpecName: "kube-api-access-kfcpj") pod "d6448e97-aa4b-4490-9039-9c377902ab55" (UID: "d6448e97-aa4b-4490-9039-9c377902ab55"). InnerVolumeSpecName "kube-api-access-kfcpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:03:13.828611 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:13.828594 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6448e97-aa4b-4490-9039-9c377902ab55-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d6448e97-aa4b-4490-9039-9c377902ab55" (UID: "d6448e97-aa4b-4490-9039-9c377902ab55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:03:13.927606 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:13.927570 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6448e97-aa4b-4490-9039-9c377902ab55-openshift-service-ca\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:03:13.927606 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:13.927599 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6448e97-aa4b-4490-9039-9c377902ab55-maas-api-tls\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:03:13.927606 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:13.927608 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfcpj\" (UniqueName: \"kubernetes.io/projected/d6448e97-aa4b-4490-9039-9c377902ab55-kube-api-access-kfcpj\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:03:14.345933 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:14.345895 2573 generic.go:356] "Generic (PLEG): container finished" podID="d6448e97-aa4b-4490-9039-9c377902ab55" containerID="e4918f9fa2ed14bc48e4cef46336860bd851f4e0a05812c1a8742db88c2497e5" exitCode=0 Jul 05 20:03:14.346359 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:14.345944 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" Jul 05 20:03:14.346359 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:14.345977 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" event={"ID":"d6448e97-aa4b-4490-9039-9c377902ab55","Type":"ContainerDied","Data":"e4918f9fa2ed14bc48e4cef46336860bd851f4e0a05812c1a8742db88c2497e5"} Jul 05 20:03:14.346359 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:14.346032 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z" event={"ID":"d6448e97-aa4b-4490-9039-9c377902ab55","Type":"ContainerDied","Data":"a49e0af57b80b322111a5e6688a434ea4a2e5a9997538253d18a249fd1aaabb3"} Jul 05 20:03:14.346359 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:14.346067 2573 scope.go:117] "RemoveContainer" containerID="e4918f9fa2ed14bc48e4cef46336860bd851f4e0a05812c1a8742db88c2497e5" Jul 05 20:03:14.355911 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:14.355892 2573 scope.go:117] "RemoveContainer" containerID="e4918f9fa2ed14bc48e4cef46336860bd851f4e0a05812c1a8742db88c2497e5" Jul 05 20:03:14.356182 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:03:14.356163 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4918f9fa2ed14bc48e4cef46336860bd851f4e0a05812c1a8742db88c2497e5\": container with ID starting with e4918f9fa2ed14bc48e4cef46336860bd851f4e0a05812c1a8742db88c2497e5 not found: ID does not exist" containerID="e4918f9fa2ed14bc48e4cef46336860bd851f4e0a05812c1a8742db88c2497e5" Jul 05 20:03:14.356249 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:14.356189 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4918f9fa2ed14bc48e4cef46336860bd851f4e0a05812c1a8742db88c2497e5"} err="failed to get container status \"e4918f9fa2ed14bc48e4cef46336860bd851f4e0a05812c1a8742db88c2497e5\": rpc error: code = NotFound desc = could not find container \"e4918f9fa2ed14bc48e4cef46336860bd851f4e0a05812c1a8742db88c2497e5\": container with ID starting with e4918f9fa2ed14bc48e4cef46336860bd851f4e0a05812c1a8742db88c2497e5 not found: ID does not exist" Jul 05 20:03:14.367515 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:14.367490 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z"] Jul 05 20:03:14.371548 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:14.371524 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-6354d0-c797b5bdb-hwt5z"] Jul 05 20:03:15.469448 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:15.469410 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6448e97-aa4b-4490-9039-9c377902ab55" path="/var/lib/kubelet/pods/d6448e97-aa4b-4490-9039-9c377902ab55/volumes" Jul 05 20:03:18.466960 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:18.466906 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:03:55.223719 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:55.223684 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:03:56.425672 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:56.425637 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l"] Jul 05 20:03:56.426184 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:56.425917 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" podUID="0e20713c-6303-4028-9183-349302235ab9" containerName="maas-api" containerID="cri-o://b7efdd9596f7652d45971df104392f64b612b8d73db0e7218268ae0f96f224d7" gracePeriod=30 Jul 05 20:03:56.523379 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:56.523346 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:03:56.551750 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:56.551717 2573 generic.go:356] "Generic (PLEG): container finished" podID="0e20713c-6303-4028-9183-349302235ab9" containerID="b7efdd9596f7652d45971df104392f64b612b8d73db0e7218268ae0f96f224d7" exitCode=0 Jul 05 20:03:56.551923 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:56.551788 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" event={"ID":"0e20713c-6303-4028-9183-349302235ab9","Type":"ContainerDied","Data":"b7efdd9596f7652d45971df104392f64b612b8d73db0e7218268ae0f96f224d7"} Jul 05 20:03:56.676097 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:56.676027 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:03:56.726748 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:56.726716 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e20713c-6303-4028-9183-349302235ab9-maas-api-tls\") pod \"0e20713c-6303-4028-9183-349302235ab9\" (UID: \"0e20713c-6303-4028-9183-349302235ab9\") " Jul 05 20:03:56.726944 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:56.726770 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw6r2\" (UniqueName: \"kubernetes.io/projected/0e20713c-6303-4028-9183-349302235ab9-kube-api-access-dw6r2\") pod \"0e20713c-6303-4028-9183-349302235ab9\" (UID: \"0e20713c-6303-4028-9183-349302235ab9\") " Jul 05 20:03:56.726944 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:56.726824 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e20713c-6303-4028-9183-349302235ab9-openshift-service-ca\") pod \"0e20713c-6303-4028-9183-349302235ab9\" (UID: \"0e20713c-6303-4028-9183-349302235ab9\") " Jul 05 20:03:56.727286 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:56.727260 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e20713c-6303-4028-9183-349302235ab9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e20713c-6303-4028-9183-349302235ab9" (UID: "0e20713c-6303-4028-9183-349302235ab9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 20:03:56.729157 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:56.729132 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e20713c-6303-4028-9183-349302235ab9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e20713c-6303-4028-9183-349302235ab9" (UID: "0e20713c-6303-4028-9183-349302235ab9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:03:56.729157 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:56.729136 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e20713c-6303-4028-9183-349302235ab9-kube-api-access-dw6r2" (OuterVolumeSpecName: "kube-api-access-dw6r2") pod "0e20713c-6303-4028-9183-349302235ab9" (UID: "0e20713c-6303-4028-9183-349302235ab9"). InnerVolumeSpecName "kube-api-access-dw6r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:03:56.828182 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:56.828146 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e20713c-6303-4028-9183-349302235ab9-maas-api-tls\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:03:56.828182 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:56.828179 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw6r2\" (UniqueName: \"kubernetes.io/projected/0e20713c-6303-4028-9183-349302235ab9-kube-api-access-dw6r2\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:03:56.828182 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:56.828189 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e20713c-6303-4028-9183-349302235ab9-openshift-service-ca\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:03:57.560744 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:57.560718 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" Jul 05 20:03:57.561205 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:57.560749 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l" event={"ID":"0e20713c-6303-4028-9183-349302235ab9","Type":"ContainerDied","Data":"f39bc028aff6bde71af1ffb9f0f1fc3dc2a5db3c60feceb5102650cf30ef7b5c"} Jul 05 20:03:57.561205 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:57.560814 2573 scope.go:117] "RemoveContainer" containerID="b7efdd9596f7652d45971df104392f64b612b8d73db0e7218268ae0f96f224d7" Jul 05 20:03:57.577654 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:57.577625 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l"] Jul 05 20:03:57.581089 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:57.581065 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-82d890-8b6495c8d-txj2l"] Jul 05 20:03:59.469055 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:03:59.469007 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e20713c-6303-4028-9183-349302235ab9" path="/var/lib/kubelet/pods/0e20713c-6303-4028-9183-349302235ab9/volumes" Jul 05 20:04:02.034542 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:02.034512 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:02.134996 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:02.134960 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:02.286544 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:02.286457 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:02.501971 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:02.501936 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:02.831824 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:02.831793 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:03.148655 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:03.148566 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:03.244502 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:03.244472 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:06.447640 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.447599 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:06.508877 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.508838 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45"] Jul 05 20:04:06.509501 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.509479 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e20713c-6303-4028-9183-349302235ab9" containerName="maas-api" Jul 05 20:04:06.509582 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.509505 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e20713c-6303-4028-9183-349302235ab9" containerName="maas-api" Jul 05 20:04:06.509582 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.509532 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6448e97-aa4b-4490-9039-9c377902ab55" containerName="maas-api" Jul 05 20:04:06.509582 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.509540 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6448e97-aa4b-4490-9039-9c377902ab55" containerName="maas-api" Jul 05 20:04:06.509741 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.509635 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e20713c-6303-4028-9183-349302235ab9" containerName="maas-api" Jul 05 20:04:06.509741 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.509647 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6448e97-aa4b-4490-9039-9c377902ab55" containerName="maas-api" Jul 05 20:04:06.513140 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.513120 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" Jul 05 20:04:06.515884 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.515864 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-a0ce36-serving-cert\"" Jul 05 20:04:06.523453 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.523430 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45"] Jul 05 20:04:06.528875 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.528854 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:06.599286 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.599263 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:06.616430 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.616408 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv8pc\" (UniqueName: \"kubernetes.io/projected/2ab7b920-4ae4-41e8-a7f7-068887568008-kube-api-access-wv8pc\") pod \"maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45\" (UID: \"2ab7b920-4ae4-41e8-a7f7-068887568008\") " pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" Jul 05 20:04:06.616537 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.616443 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ab7b920-4ae4-41e8-a7f7-068887568008-maas-api-tls\") pod \"maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45\" (UID: \"2ab7b920-4ae4-41e8-a7f7-068887568008\") " pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" Jul 05 20:04:06.616537 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.616477 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab7b920-4ae4-41e8-a7f7-068887568008-openshift-service-ca\") pod \"maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45\" (UID: \"2ab7b920-4ae4-41e8-a7f7-068887568008\") " pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" Jul 05 20:04:06.698980 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.698920 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:06.717531 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.717512 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv8pc\" (UniqueName: \"kubernetes.io/projected/2ab7b920-4ae4-41e8-a7f7-068887568008-kube-api-access-wv8pc\") pod \"maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45\" (UID: \"2ab7b920-4ae4-41e8-a7f7-068887568008\") " pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" Jul 05 20:04:06.717613 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.717545 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ab7b920-4ae4-41e8-a7f7-068887568008-maas-api-tls\") pod \"maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45\" (UID: \"2ab7b920-4ae4-41e8-a7f7-068887568008\") " pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" Jul 05 20:04:06.717613 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.717579 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab7b920-4ae4-41e8-a7f7-068887568008-openshift-service-ca\") pod \"maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45\" (UID: \"2ab7b920-4ae4-41e8-a7f7-068887568008\") " pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" Jul 05 20:04:06.718308 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.718291 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab7b920-4ae4-41e8-a7f7-068887568008-openshift-service-ca\") pod \"maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45\" (UID: \"2ab7b920-4ae4-41e8-a7f7-068887568008\") " pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" Jul 05 20:04:06.720237 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.720213 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ab7b920-4ae4-41e8-a7f7-068887568008-maas-api-tls\") pod \"maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45\" (UID: \"2ab7b920-4ae4-41e8-a7f7-068887568008\") " pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" Jul 05 20:04:06.725722 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.725703 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv8pc\" (UniqueName: \"kubernetes.io/projected/2ab7b920-4ae4-41e8-a7f7-068887568008-kube-api-access-wv8pc\") pod \"maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45\" (UID: \"2ab7b920-4ae4-41e8-a7f7-068887568008\") " pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" Jul 05 20:04:06.824868 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.824832 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" Jul 05 20:04:06.967347 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.967318 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45"] Jul 05 20:04:06.969092 ip-10-0-138-40 kubenswrapper[2573]: W0705 20:04:06.969033 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab7b920_4ae4_41e8_a7f7_068887568008.slice/crio-ae6cc768c64c0bd213f97570da0e89e9ad81cf1d6245e9d628076582970a56eb WatchSource:0}: Error finding container ae6cc768c64c0bd213f97570da0e89e9ad81cf1d6245e9d628076582970a56eb: Status 404 returned error can't find the container with id ae6cc768c64c0bd213f97570da0e89e9ad81cf1d6245e9d628076582970a56eb Jul 05 20:04:06.996193 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:06.996162 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:07.092469 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:07.092417 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:07.613414 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:07.613380 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" event={"ID":"2ab7b920-4ae4-41e8-a7f7-068887568008","Type":"ContainerStarted","Data":"2bc1ebe7f22dbe9a472c733b2b7838c5349370f05d4ab4475afcb69360096c90"} Jul 05 20:04:07.613775 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:07.613519 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" Jul 05 20:04:07.613775 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:07.613567 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" event={"ID":"2ab7b920-4ae4-41e8-a7f7-068887568008","Type":"ContainerStarted","Data":"ae6cc768c64c0bd213f97570da0e89e9ad81cf1d6245e9d628076582970a56eb"} Jul 05 20:04:07.632548 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:07.632451 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" podStartSLOduration=1.228147371 podStartE2EDuration="1.632436664s" totalImagesPullingTime="404.289293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:04:07.630567902 +0000 UTC m=+2048.630586005" watchObservedRunningTime="2026-07-05 20:04:07.632436664 +0000 UTC m=+2048.632454769" Jul 05 20:04:10.954516 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:10.954485 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:11.387702 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:11.387645 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:11.687783 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:11.687704 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:11.956139 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:11.956052 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:13.627672 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:13.627634 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" Jul 05 20:04:14.444007 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.443968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:14.501953 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.500978 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp"] Jul 05 20:04:14.505149 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.505122 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:14.507683 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.507664 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-c7ee33-serving-cert\"" Jul 05 20:04:14.514404 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.514072 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp"] Jul 05 20:04:14.529252 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.529228 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:14.590880 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.590849 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02397f6-fdaa-4bb4-b673-252b9a099fc7-maas-api-tls\") pod \"maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp\" (UID: \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\") " pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:14.591014 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.590969 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02397f6-fdaa-4bb4-b673-252b9a099fc7-openshift-service-ca\") pod \"maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp\" (UID: \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\") " pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:14.591120 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.591062 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mz6t\" (UniqueName: \"kubernetes.io/projected/d02397f6-fdaa-4bb4-b673-252b9a099fc7-kube-api-access-8mz6t\") pod \"maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp\" (UID: \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\") " pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:14.598720 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.598700 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:14.691591 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.691557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mz6t\" (UniqueName: \"kubernetes.io/projected/d02397f6-fdaa-4bb4-b673-252b9a099fc7-kube-api-access-8mz6t\") pod \"maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp\" (UID: \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\") " pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:14.691964 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.691663 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02397f6-fdaa-4bb4-b673-252b9a099fc7-maas-api-tls\") pod \"maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp\" (UID: \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\") " pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:14.691964 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.691699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02397f6-fdaa-4bb4-b673-252b9a099fc7-openshift-service-ca\") pod \"maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp\" (UID: \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\") " pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:14.691964 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:04:14.691796 2573 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-c7ee33-serving-cert: secret "maas-api-e2e-api-b-c7ee33-serving-cert" not found Jul 05 20:04:14.691964 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:04:14.691854 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d02397f6-fdaa-4bb4-b673-252b9a099fc7-maas-api-tls podName:d02397f6-fdaa-4bb4-b673-252b9a099fc7 nodeName:}" failed. No retries permitted until 2026-07-05 20:04:15.191838906 +0000 UTC m=+2056.191856991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d02397f6-fdaa-4bb4-b673-252b9a099fc7-maas-api-tls") pod "maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" (UID: "d02397f6-fdaa-4bb4-b673-252b9a099fc7") : secret "maas-api-e2e-api-b-c7ee33-serving-cert" not found Jul 05 20:04:14.692484 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.692461 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02397f6-fdaa-4bb4-b673-252b9a099fc7-openshift-service-ca\") pod \"maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp\" (UID: \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\") " pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:14.725267 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.725216 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mz6t\" (UniqueName: \"kubernetes.io/projected/d02397f6-fdaa-4bb4-b673-252b9a099fc7-kube-api-access-8mz6t\") pod \"maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp\" (UID: \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\") " pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:14.793724 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:14.793701 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:15.126143 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:15.126117 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:15.196077 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:15.196027 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02397f6-fdaa-4bb4-b673-252b9a099fc7-maas-api-tls\") pod \"maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp\" (UID: \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\") " pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:15.198432 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:15.198410 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02397f6-fdaa-4bb4-b673-252b9a099fc7-maas-api-tls\") pod \"maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp\" (UID: \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\") " pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:15.227104 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:15.227036 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:15.420773 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:15.420461 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:15.560190 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:15.560166 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp"] Jul 05 20:04:15.562434 ip-10-0-138-40 kubenswrapper[2573]: W0705 20:04:15.562407 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02397f6_fdaa_4bb4_b673_252b9a099fc7.slice/crio-96c0e15d8f0172b7983bc0fcc59778855f058d444305e338f3422b5b52b999ff WatchSource:0}: Error finding container 96c0e15d8f0172b7983bc0fcc59778855f058d444305e338f3422b5b52b999ff: Status 404 returned error can't find the container with id 96c0e15d8f0172b7983bc0fcc59778855f058d444305e338f3422b5b52b999ff Jul 05 20:04:15.659280 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:15.659245 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" event={"ID":"d02397f6-fdaa-4bb4-b673-252b9a099fc7","Type":"ContainerStarted","Data":"96c0e15d8f0172b7983bc0fcc59778855f058d444305e338f3422b5b52b999ff"} Jul 05 20:04:16.664959 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:16.664927 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" event={"ID":"d02397f6-fdaa-4bb4-b673-252b9a099fc7","Type":"ContainerStarted","Data":"d87a5f37a7f6c21b5edcd963da8b2fafdea036bb42768e3faec4ab643b49b270"} Jul 05 20:04:16.665386 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:16.665075 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:16.682797 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:16.682756 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" podStartSLOduration=2.400798714 podStartE2EDuration="2.682743062s" totalImagesPullingTime="281.944348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:04:16.680499661 +0000 UTC m=+2057.680517773" watchObservedRunningTime="2026-07-05 20:04:16.682743062 +0000 UTC m=+2057.682761168" Jul 05 20:04:18.926413 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:18.926372 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:22.677373 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:22.677304 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:26.891750 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:26.891722 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:27.296978 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:27.296944 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:27.437843 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:27.437811 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:27.552821 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:27.552738 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:27.709359 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:27.709324 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45"] Jul 05 20:04:27.709566 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:27.709527 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" podUID="2ab7b920-4ae4-41e8-a7f7-068887568008" containerName="maas-api" containerID="cri-o://2bc1ebe7f22dbe9a472c733b2b7838c5349370f05d4ab4475afcb69360096c90" gracePeriod=30 Jul 05 20:04:27.730348 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:27.730301 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:27.952710 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:27.952685 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" Jul 05 20:04:28.112143 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.112068 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ab7b920-4ae4-41e8-a7f7-068887568008-maas-api-tls\") pod \"2ab7b920-4ae4-41e8-a7f7-068887568008\" (UID: \"2ab7b920-4ae4-41e8-a7f7-068887568008\") " Jul 05 20:04:28.112143 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.112120 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv8pc\" (UniqueName: \"kubernetes.io/projected/2ab7b920-4ae4-41e8-a7f7-068887568008-kube-api-access-wv8pc\") pod \"2ab7b920-4ae4-41e8-a7f7-068887568008\" (UID: \"2ab7b920-4ae4-41e8-a7f7-068887568008\") " Jul 05 20:04:28.112376 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.112184 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab7b920-4ae4-41e8-a7f7-068887568008-openshift-service-ca\") pod \"2ab7b920-4ae4-41e8-a7f7-068887568008\" (UID: \"2ab7b920-4ae4-41e8-a7f7-068887568008\") " Jul 05 20:04:28.112612 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.112531 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab7b920-4ae4-41e8-a7f7-068887568008-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2ab7b920-4ae4-41e8-a7f7-068887568008" (UID: "2ab7b920-4ae4-41e8-a7f7-068887568008"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 20:04:28.114111 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.114086 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab7b920-4ae4-41e8-a7f7-068887568008-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2ab7b920-4ae4-41e8-a7f7-068887568008" (UID: "2ab7b920-4ae4-41e8-a7f7-068887568008"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:04:28.114279 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.114256 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab7b920-4ae4-41e8-a7f7-068887568008-kube-api-access-wv8pc" (OuterVolumeSpecName: "kube-api-access-wv8pc") pod "2ab7b920-4ae4-41e8-a7f7-068887568008" (UID: "2ab7b920-4ae4-41e8-a7f7-068887568008"). InnerVolumeSpecName "kube-api-access-wv8pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:04:28.212958 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.212916 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab7b920-4ae4-41e8-a7f7-068887568008-openshift-service-ca\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:04:28.212958 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.212955 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ab7b920-4ae4-41e8-a7f7-068887568008-maas-api-tls\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:04:28.212958 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.212970 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv8pc\" (UniqueName: \"kubernetes.io/projected/2ab7b920-4ae4-41e8-a7f7-068887568008-kube-api-access-wv8pc\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:04:28.735694 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.735656 2573 generic.go:356] "Generic (PLEG): container finished" podID="2ab7b920-4ae4-41e8-a7f7-068887568008" containerID="2bc1ebe7f22dbe9a472c733b2b7838c5349370f05d4ab4475afcb69360096c90" exitCode=0 Jul 05 20:04:28.735878 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.735702 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" Jul 05 20:04:28.735878 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.735750 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" event={"ID":"2ab7b920-4ae4-41e8-a7f7-068887568008","Type":"ContainerDied","Data":"2bc1ebe7f22dbe9a472c733b2b7838c5349370f05d4ab4475afcb69360096c90"} Jul 05 20:04:28.735878 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.735807 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45" event={"ID":"2ab7b920-4ae4-41e8-a7f7-068887568008","Type":"ContainerDied","Data":"ae6cc768c64c0bd213f97570da0e89e9ad81cf1d6245e9d628076582970a56eb"} Jul 05 20:04:28.735878 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.735837 2573 scope.go:117] "RemoveContainer" containerID="2bc1ebe7f22dbe9a472c733b2b7838c5349370f05d4ab4475afcb69360096c90" Jul 05 20:04:28.745197 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.745179 2573 scope.go:117] "RemoveContainer" containerID="2bc1ebe7f22dbe9a472c733b2b7838c5349370f05d4ab4475afcb69360096c90" Jul 05 20:04:28.745463 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:04:28.745444 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc1ebe7f22dbe9a472c733b2b7838c5349370f05d4ab4475afcb69360096c90\": container with ID starting with 2bc1ebe7f22dbe9a472c733b2b7838c5349370f05d4ab4475afcb69360096c90 not found: ID does not exist" containerID="2bc1ebe7f22dbe9a472c733b2b7838c5349370f05d4ab4475afcb69360096c90" Jul 05 20:04:28.745516 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.745473 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc1ebe7f22dbe9a472c733b2b7838c5349370f05d4ab4475afcb69360096c90"} err="failed to get container status \"2bc1ebe7f22dbe9a472c733b2b7838c5349370f05d4ab4475afcb69360096c90\": rpc error: code = NotFound desc = could not find container \"2bc1ebe7f22dbe9a472c733b2b7838c5349370f05d4ab4475afcb69360096c90\": container with ID starting with 2bc1ebe7f22dbe9a472c733b2b7838c5349370f05d4ab4475afcb69360096c90 not found: ID does not exist" Jul 05 20:04:28.758225 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.758193 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45"] Jul 05 20:04:28.766008 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:28.762435 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-a0ce36-5dc876d999-kfh45"] Jul 05 20:04:29.471008 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:29.470973 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ab7b920-4ae4-41e8-a7f7-068887568008" path="/var/lib/kubelet/pods/2ab7b920-4ae4-41e8-a7f7-068887568008/volumes" Jul 05 20:04:33.078864 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:33.078824 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:33.178716 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:33.178664 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:33.320802 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:33.320766 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:33.552700 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:33.552654 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:33.707780 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:33.707723 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:34.084082 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.084007 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:34.184468 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.184443 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:34.291846 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.291807 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp"] Jul 05 20:04:34.292061 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.292019 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" podUID="d02397f6-fdaa-4bb4-b673-252b9a099fc7" containerName="maas-api" containerID="cri-o://d87a5f37a7f6c21b5edcd963da8b2fafdea036bb42768e3faec4ab643b49b270" gracePeriod=30 Jul 05 20:04:34.315470 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.315420 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:34.538639 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.538616 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:34.569930 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.569900 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02397f6-fdaa-4bb4-b673-252b9a099fc7-maas-api-tls\") pod \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\" (UID: \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\") " Jul 05 20:04:34.570086 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.569952 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mz6t\" (UniqueName: \"kubernetes.io/projected/d02397f6-fdaa-4bb4-b673-252b9a099fc7-kube-api-access-8mz6t\") pod \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\" (UID: \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\") " Jul 05 20:04:34.570086 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.570007 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02397f6-fdaa-4bb4-b673-252b9a099fc7-openshift-service-ca\") pod \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\" (UID: \"d02397f6-fdaa-4bb4-b673-252b9a099fc7\") " Jul 05 20:04:34.570460 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.570432 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02397f6-fdaa-4bb4-b673-252b9a099fc7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d02397f6-fdaa-4bb4-b673-252b9a099fc7" (UID: "d02397f6-fdaa-4bb4-b673-252b9a099fc7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 20:04:34.572084 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.572065 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d02397f6-fdaa-4bb4-b673-252b9a099fc7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d02397f6-fdaa-4bb4-b673-252b9a099fc7" (UID: "d02397f6-fdaa-4bb4-b673-252b9a099fc7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:04:34.572230 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.572207 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02397f6-fdaa-4bb4-b673-252b9a099fc7-kube-api-access-8mz6t" (OuterVolumeSpecName: "kube-api-access-8mz6t") pod "d02397f6-fdaa-4bb4-b673-252b9a099fc7" (UID: "d02397f6-fdaa-4bb4-b673-252b9a099fc7"). InnerVolumeSpecName "kube-api-access-8mz6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:04:34.671296 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.671191 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d02397f6-fdaa-4bb4-b673-252b9a099fc7-openshift-service-ca\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:04:34.671296 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.671236 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d02397f6-fdaa-4bb4-b673-252b9a099fc7-maas-api-tls\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:04:34.671296 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.671249 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mz6t\" (UniqueName: \"kubernetes.io/projected/d02397f6-fdaa-4bb4-b673-252b9a099fc7-kube-api-access-8mz6t\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:04:34.767182 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.767141 2573 generic.go:356] "Generic (PLEG): container finished" podID="d02397f6-fdaa-4bb4-b673-252b9a099fc7" containerID="d87a5f37a7f6c21b5edcd963da8b2fafdea036bb42768e3faec4ab643b49b270" exitCode=0 Jul 05 20:04:34.767325 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.767192 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" Jul 05 20:04:34.767325 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.767214 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" event={"ID":"d02397f6-fdaa-4bb4-b673-252b9a099fc7","Type":"ContainerDied","Data":"d87a5f37a7f6c21b5edcd963da8b2fafdea036bb42768e3faec4ab643b49b270"} Jul 05 20:04:34.767325 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.767301 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp" event={"ID":"d02397f6-fdaa-4bb4-b673-252b9a099fc7","Type":"ContainerDied","Data":"96c0e15d8f0172b7983bc0fcc59778855f058d444305e338f3422b5b52b999ff"} Jul 05 20:04:34.767453 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.767327 2573 scope.go:117] "RemoveContainer" containerID="d87a5f37a7f6c21b5edcd963da8b2fafdea036bb42768e3faec4ab643b49b270" Jul 05 20:04:34.776152 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.776134 2573 scope.go:117] "RemoveContainer" containerID="d87a5f37a7f6c21b5edcd963da8b2fafdea036bb42768e3faec4ab643b49b270" Jul 05 20:04:34.776431 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:04:34.776413 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d87a5f37a7f6c21b5edcd963da8b2fafdea036bb42768e3faec4ab643b49b270\": container with ID starting with d87a5f37a7f6c21b5edcd963da8b2fafdea036bb42768e3faec4ab643b49b270 not found: ID does not exist" containerID="d87a5f37a7f6c21b5edcd963da8b2fafdea036bb42768e3faec4ab643b49b270" Jul 05 20:04:34.776509 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.776435 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d87a5f37a7f6c21b5edcd963da8b2fafdea036bb42768e3faec4ab643b49b270"} err="failed to get container status \"d87a5f37a7f6c21b5edcd963da8b2fafdea036bb42768e3faec4ab643b49b270\": rpc error: code = NotFound desc = could not find container \"d87a5f37a7f6c21b5edcd963da8b2fafdea036bb42768e3faec4ab643b49b270\": container with ID starting with d87a5f37a7f6c21b5edcd963da8b2fafdea036bb42768e3faec4ab643b49b270 not found: ID does not exist" Jul 05 20:04:34.788655 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.788626 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp"] Jul 05 20:04:34.791841 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:34.791816 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-c7ee33-67b4b4d788-b6mfp"] Jul 05 20:04:35.468900 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:35.468864 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d02397f6-fdaa-4bb4-b673-252b9a099fc7" path="/var/lib/kubelet/pods/d02397f6-fdaa-4bb4-b673-252b9a099fc7/volumes" Jul 05 20:04:36.534253 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:36.534226 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:36.634418 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:36.634390 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:39.684321 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:39.684285 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:39.791776 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:39.791740 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:40.012842 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:40.012806 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:40.165473 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:40.165441 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:40.491854 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:40.491768 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:40.782516 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:40.782481 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:40.965063 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:40.965008 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:43.615480 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.615432 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:43.717358 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.717324 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:43.783965 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.783931 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2"] Jul 05 20:04:43.784618 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.784598 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d02397f6-fdaa-4bb4-b673-252b9a099fc7" containerName="maas-api" Jul 05 20:04:43.784666 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.784624 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d02397f6-fdaa-4bb4-b673-252b9a099fc7" containerName="maas-api" Jul 05 20:04:43.784666 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.784655 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ab7b920-4ae4-41e8-a7f7-068887568008" containerName="maas-api" Jul 05 20:04:43.784666 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.784664 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ab7b920-4ae4-41e8-a7f7-068887568008" containerName="maas-api" Jul 05 20:04:43.784774 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.784761 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d02397f6-fdaa-4bb4-b673-252b9a099fc7" containerName="maas-api" Jul 05 20:04:43.784811 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.784782 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ab7b920-4ae4-41e8-a7f7-068887568008" containerName="maas-api" Jul 05 20:04:43.789804 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.789782 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:04:43.792768 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.792750 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-ec8ab7-serving-cert\"" Jul 05 20:04:43.797111 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.797078 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2"] Jul 05 20:04:43.813150 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.813124 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:43.846699 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.846672 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2w25\" (UniqueName: \"kubernetes.io/projected/0d6eca4d-207a-4154-9085-322fd166d13a-kube-api-access-p2w25\") pod \"maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2\" (UID: \"0d6eca4d-207a-4154-9085-322fd166d13a\") " pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:04:43.846808 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.846708 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d6eca4d-207a-4154-9085-322fd166d13a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2\" (UID: \"0d6eca4d-207a-4154-9085-322fd166d13a\") " pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:04:43.846850 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.846811 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d6eca4d-207a-4154-9085-322fd166d13a-maas-api-tls\") pod \"maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2\" (UID: \"0d6eca4d-207a-4154-9085-322fd166d13a\") " pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:04:43.947660 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.947586 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d6eca4d-207a-4154-9085-322fd166d13a-maas-api-tls\") pod \"maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2\" (UID: \"0d6eca4d-207a-4154-9085-322fd166d13a\") " pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:04:43.947768 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.947662 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2w25\" (UniqueName: \"kubernetes.io/projected/0d6eca4d-207a-4154-9085-322fd166d13a-kube-api-access-p2w25\") pod \"maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2\" (UID: \"0d6eca4d-207a-4154-9085-322fd166d13a\") " pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:04:43.947768 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.947682 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d6eca4d-207a-4154-9085-322fd166d13a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2\" (UID: \"0d6eca4d-207a-4154-9085-322fd166d13a\") " pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:04:43.947768 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:04:43.947731 2573 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-ec8ab7-serving-cert: secret "maas-api-e2e-shared-a-ec8ab7-serving-cert" not found Jul 05 20:04:43.947901 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:04:43.947801 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d6eca4d-207a-4154-9085-322fd166d13a-maas-api-tls podName:0d6eca4d-207a-4154-9085-322fd166d13a nodeName:}" failed. No retries permitted until 2026-07-05 20:04:44.447781772 +0000 UTC m=+2085.447799856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d6eca4d-207a-4154-9085-322fd166d13a-maas-api-tls") pod "maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" (UID: "0d6eca4d-207a-4154-9085-322fd166d13a") : secret "maas-api-e2e-shared-a-ec8ab7-serving-cert" not found Jul 05 20:04:43.948319 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.948303 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d6eca4d-207a-4154-9085-322fd166d13a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2\" (UID: \"0d6eca4d-207a-4154-9085-322fd166d13a\") " pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:04:43.958259 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:43.958239 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2w25\" (UniqueName: \"kubernetes.io/projected/0d6eca4d-207a-4154-9085-322fd166d13a-kube-api-access-p2w25\") pod \"maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2\" (UID: \"0d6eca4d-207a-4154-9085-322fd166d13a\") " pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:04:44.088591 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:44.088564 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:44.342076 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:44.342036 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:44.442119 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:44.442098 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:44.452163 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:44.452143 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d6eca4d-207a-4154-9085-322fd166d13a-maas-api-tls\") pod \"maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2\" (UID: \"0d6eca4d-207a-4154-9085-322fd166d13a\") " pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:04:44.454652 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:44.454627 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d6eca4d-207a-4154-9085-322fd166d13a-maas-api-tls\") pod \"maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2\" (UID: \"0d6eca4d-207a-4154-9085-322fd166d13a\") " pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:04:44.702437 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:44.702380 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:04:44.844432 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:44.844402 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2"] Jul 05 20:04:44.848509 ip-10-0-138-40 kubenswrapper[2573]: W0705 20:04:44.848479 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6eca4d_207a_4154_9085_322fd166d13a.slice/crio-c20887c2c86447574c880beff7c050b9afbb2ea83b9fd13d2c497e6cf2ed8906 WatchSource:0}: Error finding container c20887c2c86447574c880beff7c050b9afbb2ea83b9fd13d2c497e6cf2ed8906: Status 404 returned error can't find the container with id c20887c2c86447574c880beff7c050b9afbb2ea83b9fd13d2c497e6cf2ed8906 Jul 05 20:04:45.827488 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:45.827447 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" event={"ID":"0d6eca4d-207a-4154-9085-322fd166d13a","Type":"ContainerStarted","Data":"5159016c9d1ec2779584ecfa282bcd77a88795d6f43a6fca2916ead961db0300"} Jul 05 20:04:45.827944 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:45.827511 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" event={"ID":"0d6eca4d-207a-4154-9085-322fd166d13a","Type":"ContainerStarted","Data":"c20887c2c86447574c880beff7c050b9afbb2ea83b9fd13d2c497e6cf2ed8906"} Jul 05 20:04:45.827944 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:45.827568 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:04:45.844849 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:45.844801 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" podStartSLOduration=2.550012552 podStartE2EDuration="2.844788486s" totalImagesPullingTime="294.775934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:04:45.842947642 +0000 UTC m=+2086.842965743" watchObservedRunningTime="2026-07-05 20:04:45.844788486 +0000 UTC m=+2086.844806591" Jul 05 20:04:48.119135 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:48.119103 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:48.485134 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:48.485060 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:48.750680 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:48.750646 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:48.896827 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:48.896793 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:51.661631 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.661598 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:51.715314 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.715285 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45"] Jul 05 20:04:51.726852 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.726807 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:04:51.729362 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.728778 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45"] Jul 05 20:04:51.730111 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.730080 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-c11b28-serving-cert\"" Jul 05 20:04:51.745773 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.745747 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:51.796464 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.796445 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:51.813111 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.813082 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdt9g\" (UniqueName: \"kubernetes.io/projected/6f0bc2ee-2501-4225-85ba-33f400601f05-kube-api-access-pdt9g\") pod \"maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45\" (UID: \"6f0bc2ee-2501-4225-85ba-33f400601f05\") " pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:04:51.813202 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.813131 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f0bc2ee-2501-4225-85ba-33f400601f05-maas-api-tls\") pod \"maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45\" (UID: \"6f0bc2ee-2501-4225-85ba-33f400601f05\") " pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:04:51.813202 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.813184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0bc2ee-2501-4225-85ba-33f400601f05-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45\" (UID: \"6f0bc2ee-2501-4225-85ba-33f400601f05\") " pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:04:51.837613 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.837587 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:04:51.914193 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.914138 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdt9g\" (UniqueName: \"kubernetes.io/projected/6f0bc2ee-2501-4225-85ba-33f400601f05-kube-api-access-pdt9g\") pod \"maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45\" (UID: \"6f0bc2ee-2501-4225-85ba-33f400601f05\") " pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:04:51.914193 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.914177 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f0bc2ee-2501-4225-85ba-33f400601f05-maas-api-tls\") pod \"maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45\" (UID: \"6f0bc2ee-2501-4225-85ba-33f400601f05\") " pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:04:51.914373 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.914199 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0bc2ee-2501-4225-85ba-33f400601f05-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45\" (UID: \"6f0bc2ee-2501-4225-85ba-33f400601f05\") " pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:04:51.914373 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:04:51.914291 2573 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-c11b28-serving-cert: secret "maas-api-e2e-shared-b-c11b28-serving-cert" not found Jul 05 20:04:51.914373 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:04:51.914367 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f0bc2ee-2501-4225-85ba-33f400601f05-maas-api-tls podName:6f0bc2ee-2501-4225-85ba-33f400601f05 nodeName:}" failed. No retries permitted until 2026-07-05 20:04:52.414347717 +0000 UTC m=+2093.414365802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f0bc2ee-2501-4225-85ba-33f400601f05-maas-api-tls") pod "maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" (UID: "6f0bc2ee-2501-4225-85ba-33f400601f05") : secret "maas-api-e2e-shared-b-c11b28-serving-cert" not found Jul 05 20:04:51.914831 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.914812 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0bc2ee-2501-4225-85ba-33f400601f05-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45\" (UID: \"6f0bc2ee-2501-4225-85ba-33f400601f05\") " pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:04:51.923145 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:51.923116 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdt9g\" (UniqueName: \"kubernetes.io/projected/6f0bc2ee-2501-4225-85ba-33f400601f05-kube-api-access-pdt9g\") pod \"maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45\" (UID: \"6f0bc2ee-2501-4225-85ba-33f400601f05\") " pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:04:52.024726 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:52.024701 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:52.186701 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:52.186632 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:52.287726 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:52.287702 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:52.417348 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:52.417322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f0bc2ee-2501-4225-85ba-33f400601f05-maas-api-tls\") pod \"maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45\" (UID: \"6f0bc2ee-2501-4225-85ba-33f400601f05\") " pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:04:52.419825 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:52.419805 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f0bc2ee-2501-4225-85ba-33f400601f05-maas-api-tls\") pod \"maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45\" (UID: \"6f0bc2ee-2501-4225-85ba-33f400601f05\") " pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:04:52.641187 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:52.641162 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:04:52.805783 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:52.805756 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45"] Jul 05 20:04:52.808117 ip-10-0-138-40 kubenswrapper[2573]: W0705 20:04:52.808033 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0bc2ee_2501_4225_85ba_33f400601f05.slice/crio-b99b7181a3ebdeffee85095c2f988eaacaa15476bf176906d115a842bb7ac98c WatchSource:0}: Error finding container b99b7181a3ebdeffee85095c2f988eaacaa15476bf176906d115a842bb7ac98c: Status 404 returned error can't find the container with id b99b7181a3ebdeffee85095c2f988eaacaa15476bf176906d115a842bb7ac98c Jul 05 20:04:52.865024 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:52.864991 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" event={"ID":"6f0bc2ee-2501-4225-85ba-33f400601f05","Type":"ContainerStarted","Data":"b99b7181a3ebdeffee85095c2f988eaacaa15476bf176906d115a842bb7ac98c"} Jul 05 20:04:53.873032 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:53.872990 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" event={"ID":"6f0bc2ee-2501-4225-85ba-33f400601f05","Type":"ContainerStarted","Data":"702254718b0c83dd29462a3cb1886ed2e982f298357d0871139849fe06e927ff"} Jul 05 20:04:53.873517 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:53.873155 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:04:53.889833 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:53.889781 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" podStartSLOduration=2.502918294 podStartE2EDuration="2.889764252s" totalImagesPullingTime="386.845958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:04:53.888895098 +0000 UTC m=+2094.888913204" watchObservedRunningTime="2026-07-05 20:04:53.889764252 +0000 UTC m=+2094.889782359" Jul 05 20:04:56.139244 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:56.139200 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:04:59.623351 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:59.623317 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 20:04:59.629665 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:59.629639 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 20:04:59.884349 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:04:59.884288 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:05:04.948897 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:04.948868 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:05:05.049541 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:05.049508 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:05:08.942665 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:08.942633 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:05:12.821453 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:12.821423 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:05:12.920623 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:12.920591 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:05:34.242019 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.241979 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f"] Jul 05 20:05:34.245350 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.245327 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.248156 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.248128 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c11b28\"/\"kube-root-ca.crt\"" Jul 05 20:05:34.248277 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.248173 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c11b28\"/\"default-dockercfg-tx5b9\"" Jul 05 20:05:34.249711 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.249693 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c11b28\"/\"openshift-service-ca.crt\"" Jul 05 20:05:34.249711 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.249701 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c11b28\"/\"auth-test-model-c11b28-kserve-self-signed-certs\"" Jul 05 20:05:34.256216 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.256193 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f"] Jul 05 20:05:34.266355 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.266325 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6b5b\" (UniqueName: \"kubernetes.io/projected/3794047e-281c-4432-9aaa-4e19d914c039-kube-api-access-z6b5b\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.266480 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.266401 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-dshm\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.266480 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.266429 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-home\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.266480 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.266452 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-tmp-dir\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.266657 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.266548 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3794047e-281c-4432-9aaa-4e19d914c039-tls-certs\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.266657 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.266613 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-model-cache\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.266840 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.266818 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:05:34.367316 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.367290 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-home\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.367316 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.367319 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-tmp-dir\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.367537 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.367373 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3794047e-281c-4432-9aaa-4e19d914c039-tls-certs\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.367537 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.367418 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-model-cache\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.367537 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.367470 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6b5b\" (UniqueName: \"kubernetes.io/projected/3794047e-281c-4432-9aaa-4e19d914c039-kube-api-access-z6b5b\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.367537 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.367512 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-dshm\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.367742 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.367710 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-home\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.367742 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.367734 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-tmp-dir\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.367852 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.367809 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-model-cache\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.369935 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.369912 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-dshm\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.370180 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.370163 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3794047e-281c-4432-9aaa-4e19d914c039-tls-certs\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.375264 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.375243 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6b5b\" (UniqueName: \"kubernetes.io/projected/3794047e-281c-4432-9aaa-4e19d914c039-kube-api-access-z6b5b\") pod \"auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.556575 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.556549 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:34.691399 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:34.691376 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f"] Jul 05 20:05:34.693377 ip-10-0-138-40 kubenswrapper[2573]: W0705 20:05:34.693350 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3794047e_281c_4432_9aaa_4e19d914c039.slice/crio-56d00114eee35ec00ff400ab258a5831ac79bb4165660b645d2cd3ac4f432f13 WatchSource:0}: Error finding container 56d00114eee35ec00ff400ab258a5831ac79bb4165660b645d2cd3ac4f432f13: Status 404 returned error can't find the container with id 56d00114eee35ec00ff400ab258a5831ac79bb4165660b645d2cd3ac4f432f13 Jul 05 20:05:35.090033 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:35.090000 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" event={"ID":"3794047e-281c-4432-9aaa-4e19d914c039","Type":"ContainerStarted","Data":"7d3687d77caa01b24ef641f4d0273ddd9b96c86991875837b96df566535b23b0"} Jul 05 20:05:35.090203 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:35.090050 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" event={"ID":"3794047e-281c-4432-9aaa-4e19d914c039","Type":"ContainerStarted","Data":"56d00114eee35ec00ff400ab258a5831ac79bb4165660b645d2cd3ac4f432f13"} Jul 05 20:05:35.108162 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:35.108122 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" podStartSLOduration=1.108107579 podStartE2EDuration="1.108107579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:05:35.106304625 +0000 UTC m=+2136.106322730" watchObservedRunningTime="2026-07-05 20:05:35.108107579 +0000 UTC m=+2136.108125685" Jul 05 20:05:44.557170 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:44.557130 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:44.557170 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:44.557177 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:44.568727 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:44.568701 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:05:45.157411 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:05:45.157385 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:07:32.792253 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:32.792169 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:07:32.878756 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:32.878732 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:07:43.045500 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:43.045469 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:07:43.923363 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:43.923323 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f"] Jul 05 20:07:43.942307 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:43.942279 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:07:56.754135 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:56.754096 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:07:58.944563 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:58.944528 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" podUID="3794047e-281c-4432-9aaa-4e19d914c039" containerName="main" containerID="cri-o://7d3687d77caa01b24ef641f4d0273ddd9b96c86991875837b96df566535b23b0" gracePeriod=45 Jul 05 20:07:59.195679 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.195619 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:07:59.271973 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.271936 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-dshm\") pod \"3794047e-281c-4432-9aaa-4e19d914c039\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " Jul 05 20:07:59.272191 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.271990 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-model-cache\") pod \"3794047e-281c-4432-9aaa-4e19d914c039\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " Jul 05 20:07:59.272191 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.272018 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-home\") pod \"3794047e-281c-4432-9aaa-4e19d914c039\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " Jul 05 20:07:59.272191 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.272105 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6b5b\" (UniqueName: \"kubernetes.io/projected/3794047e-281c-4432-9aaa-4e19d914c039-kube-api-access-z6b5b\") pod \"3794047e-281c-4432-9aaa-4e19d914c039\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " Jul 05 20:07:59.272373 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.272269 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-tmp-dir\") pod \"3794047e-281c-4432-9aaa-4e19d914c039\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " Jul 05 20:07:59.272373 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.272281 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-model-cache" (OuterVolumeSpecName: "model-cache") pod "3794047e-281c-4432-9aaa-4e19d914c039" (UID: "3794047e-281c-4432-9aaa-4e19d914c039"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:07:59.272373 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.272313 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3794047e-281c-4432-9aaa-4e19d914c039-tls-certs\") pod \"3794047e-281c-4432-9aaa-4e19d914c039\" (UID: \"3794047e-281c-4432-9aaa-4e19d914c039\") " Jul 05 20:07:59.272373 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.272344 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-home" (OuterVolumeSpecName: "home") pod "3794047e-281c-4432-9aaa-4e19d914c039" (UID: "3794047e-281c-4432-9aaa-4e19d914c039"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:07:59.272537 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.272500 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3794047e-281c-4432-9aaa-4e19d914c039" (UID: "3794047e-281c-4432-9aaa-4e19d914c039"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:07:59.272686 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.272670 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-model-cache\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:07:59.272734 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.272689 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-home\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:07:59.272734 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.272697 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-tmp-dir\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:07:59.274559 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.274540 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3794047e-281c-4432-9aaa-4e19d914c039-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3794047e-281c-4432-9aaa-4e19d914c039" (UID: "3794047e-281c-4432-9aaa-4e19d914c039"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:07:59.274700 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.274682 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-dshm" (OuterVolumeSpecName: "dshm") pod "3794047e-281c-4432-9aaa-4e19d914c039" (UID: "3794047e-281c-4432-9aaa-4e19d914c039"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:07:59.274759 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.274702 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3794047e-281c-4432-9aaa-4e19d914c039-kube-api-access-z6b5b" (OuterVolumeSpecName: "kube-api-access-z6b5b") pod "3794047e-281c-4432-9aaa-4e19d914c039" (UID: "3794047e-281c-4432-9aaa-4e19d914c039"). InnerVolumeSpecName "kube-api-access-z6b5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:07:59.373449 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.373412 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6b5b\" (UniqueName: \"kubernetes.io/projected/3794047e-281c-4432-9aaa-4e19d914c039-kube-api-access-z6b5b\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:07:59.373449 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.373439 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3794047e-281c-4432-9aaa-4e19d914c039-tls-certs\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:07:59.373449 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.373449 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3794047e-281c-4432-9aaa-4e19d914c039-dshm\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:07:59.792444 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.792407 2573 generic.go:356] "Generic (PLEG): container finished" podID="3794047e-281c-4432-9aaa-4e19d914c039" containerID="7d3687d77caa01b24ef641f4d0273ddd9b96c86991875837b96df566535b23b0" exitCode=0 Jul 05 20:07:59.792654 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.792456 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" Jul 05 20:07:59.792654 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.792486 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" event={"ID":"3794047e-281c-4432-9aaa-4e19d914c039","Type":"ContainerDied","Data":"7d3687d77caa01b24ef641f4d0273ddd9b96c86991875837b96df566535b23b0"} Jul 05 20:07:59.792654 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.792529 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f" event={"ID":"3794047e-281c-4432-9aaa-4e19d914c039","Type":"ContainerDied","Data":"56d00114eee35ec00ff400ab258a5831ac79bb4165660b645d2cd3ac4f432f13"} Jul 05 20:07:59.792654 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.792548 2573 scope.go:117] "RemoveContainer" containerID="7d3687d77caa01b24ef641f4d0273ddd9b96c86991875837b96df566535b23b0" Jul 05 20:07:59.803615 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.803562 2573 scope.go:117] "RemoveContainer" containerID="7d3687d77caa01b24ef641f4d0273ddd9b96c86991875837b96df566535b23b0" Jul 05 20:07:59.804002 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:07:59.803982 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d3687d77caa01b24ef641f4d0273ddd9b96c86991875837b96df566535b23b0\": container with ID starting with 7d3687d77caa01b24ef641f4d0273ddd9b96c86991875837b96df566535b23b0 not found: ID does not exist" containerID="7d3687d77caa01b24ef641f4d0273ddd9b96c86991875837b96df566535b23b0" Jul 05 20:07:59.804082 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.804012 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d3687d77caa01b24ef641f4d0273ddd9b96c86991875837b96df566535b23b0"} err="failed to get container status \"7d3687d77caa01b24ef641f4d0273ddd9b96c86991875837b96df566535b23b0\": rpc error: code = NotFound desc = could not find container \"7d3687d77caa01b24ef641f4d0273ddd9b96c86991875837b96df566535b23b0\": container with ID starting with 7d3687d77caa01b24ef641f4d0273ddd9b96c86991875837b96df566535b23b0 not found: ID does not exist" Jul 05 20:07:59.809972 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.809943 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f"] Jul 05 20:07:59.811779 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:07:59.811760 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c11b28/auth-test-model-c11b28-kserve-65c7bf7c9f-76x2f"] Jul 05 20:08:01.469404 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:08:01.469370 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3794047e-281c-4432-9aaa-4e19d914c039" path="/var/lib/kubelet/pods/3794047e-281c-4432-9aaa-4e19d914c039/volumes" Jul 05 20:08:25.231166 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:08:25.231133 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:09:02.441220 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:02.441186 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:09:03.645640 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:03.645604 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:09:04.731093 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:04.731036 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:09:12.742657 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.742621 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28"] Jul 05 20:09:12.743231 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.743212 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3794047e-281c-4432-9aaa-4e19d914c039" containerName="main" Jul 05 20:09:12.743282 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.743235 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3794047e-281c-4432-9aaa-4e19d914c039" containerName="main" Jul 05 20:09:12.743384 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.743373 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3794047e-281c-4432-9aaa-4e19d914c039" containerName="main" Jul 05 20:09:12.746523 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.746506 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:12.750645 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.750602 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ec8ab7\"/\"openshift-service-ca.crt\"" Jul 05 20:09:12.750645 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.750627 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ec8ab7\"/\"rate-test-model-ec8ab7-kserve-self-signed-certs\"" Jul 05 20:09:12.750855 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.750645 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-ec8ab7\"/\"default-dockercfg-g4gcj\"" Jul 05 20:09:12.750855 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.750703 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-ec8ab7\"/\"kube-root-ca.crt\"" Jul 05 20:09:12.754034 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.753561 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28"] Jul 05 20:09:12.768898 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.768868 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:09:12.909837 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.909811 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dfgk\" (UniqueName: \"kubernetes.io/projected/a1928f92-9790-4a2e-ab50-59492eafe072-kube-api-access-7dfgk\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:12.909944 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.909843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-home\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:12.909944 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.909870 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-tmp-dir\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:12.910018 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.910005 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-dshm\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:12.910085 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.910027 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1928f92-9790-4a2e-ab50-59492eafe072-tls-certs\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:12.910125 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:12.910102 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-model-cache\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:13.011380 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:13.011355 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-dshm\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:13.011477 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:13.011385 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1928f92-9790-4a2e-ab50-59492eafe072-tls-certs\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:13.011477 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:13.011406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-model-cache\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:13.011568 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:13.011474 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dfgk\" (UniqueName: \"kubernetes.io/projected/a1928f92-9790-4a2e-ab50-59492eafe072-kube-api-access-7dfgk\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:13.011568 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:13.011503 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-home\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:13.011568 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:13.011530 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-tmp-dir\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:13.011886 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:13.011861 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-model-cache\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:13.011984 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:13.011891 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-home\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:13.011984 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:13.011920 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-tmp-dir\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:13.013769 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:13.013750 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-dshm\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:13.013864 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:13.013849 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1928f92-9790-4a2e-ab50-59492eafe072-tls-certs\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:13.019331 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:13.019307 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dfgk\" (UniqueName: \"kubernetes.io/projected/a1928f92-9790-4a2e-ab50-59492eafe072-kube-api-access-7dfgk\") pod \"rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:13.057303 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:13.057282 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:13.187890 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:13.187867 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28"] Jul 05 20:09:13.190016 ip-10-0-138-40 kubenswrapper[2573]: W0705 20:09:13.189987 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1928f92_9790_4a2e_ab50_59492eafe072.slice/crio-dbe116dd694129875333e27757bece6341d4a2f9eecae01f51706c2e5b855ba7 WatchSource:0}: Error finding container dbe116dd694129875333e27757bece6341d4a2f9eecae01f51706c2e5b855ba7: Status 404 returned error can't find the container with id dbe116dd694129875333e27757bece6341d4a2f9eecae01f51706c2e5b855ba7 Jul 05 20:09:14.187279 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:14.187237 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" event={"ID":"a1928f92-9790-4a2e-ab50-59492eafe072","Type":"ContainerStarted","Data":"62f437126c1de5dd66921565ba62cfb3546baaf2a7e51d011b762a275e5da024"} Jul 05 20:09:14.187279 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:14.187284 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" event={"ID":"a1928f92-9790-4a2e-ab50-59492eafe072","Type":"ContainerStarted","Data":"dbe116dd694129875333e27757bece6341d4a2f9eecae01f51706c2e5b855ba7"} Jul 05 20:09:14.205115 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:14.205057 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" podStartSLOduration=2.205023198 podStartE2EDuration="2.205023198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 20:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:09:14.20223229 +0000 UTC m=+2355.202250410" watchObservedRunningTime="2026-07-05 20:09:14.205023198 +0000 UTC m=+2355.205041303" Jul 05 20:09:20.228417 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:20.228379 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:09:23.058123 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:23.058071 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:23.058523 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:23.058137 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:23.071024 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:23.070996 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:23.246697 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:23.246671 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:09:37.152158 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:37.152122 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:09:37.252614 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:37.252587 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:09:38.560493 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:38.560465 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:09:59.658164 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:59.658135 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 20:09:59.663977 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:09:59.663943 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 20:10:05.575933 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:05.575899 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:10:05.677382 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:05.677351 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:10:09.455234 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:09.455195 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:10:13.525006 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:13.524968 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:10:13.623432 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:13.623406 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:10:37.823918 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:37.823838 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28"] Jul 05 20:10:37.844101 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:37.844077 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:10:38.741175 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:38.741144 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:10:39.796754 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:39.796714 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:10:40.185305 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:40.185233 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:10:40.286094 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:40.286066 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:10:40.384488 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:40.384464 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2"] Jul 05 20:10:40.384709 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:40.384684 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" podUID="0d6eca4d-207a-4154-9085-322fd166d13a" containerName="maas-api" containerID="cri-o://5159016c9d1ec2779584ecfa282bcd77a88795d6f43a6fca2916ead961db0300" gracePeriod=30 Jul 05 20:10:40.408445 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:40.408413 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:10:40.827096 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:40.827070 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:10:40.934641 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:40.934585 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d6eca4d-207a-4154-9085-322fd166d13a-maas-api-tls\") pod \"0d6eca4d-207a-4154-9085-322fd166d13a\" (UID: \"0d6eca4d-207a-4154-9085-322fd166d13a\") " Jul 05 20:10:40.934641 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:40.934630 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d6eca4d-207a-4154-9085-322fd166d13a-openshift-service-ca\") pod \"0d6eca4d-207a-4154-9085-322fd166d13a\" (UID: \"0d6eca4d-207a-4154-9085-322fd166d13a\") " Jul 05 20:10:40.934804 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:40.934723 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2w25\" (UniqueName: \"kubernetes.io/projected/0d6eca4d-207a-4154-9085-322fd166d13a-kube-api-access-p2w25\") pod \"0d6eca4d-207a-4154-9085-322fd166d13a\" (UID: \"0d6eca4d-207a-4154-9085-322fd166d13a\") " Jul 05 20:10:40.935085 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:40.935031 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6eca4d-207a-4154-9085-322fd166d13a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0d6eca4d-207a-4154-9085-322fd166d13a" (UID: "0d6eca4d-207a-4154-9085-322fd166d13a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 20:10:40.936885 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:40.936862 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d6eca4d-207a-4154-9085-322fd166d13a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0d6eca4d-207a-4154-9085-322fd166d13a" (UID: "0d6eca4d-207a-4154-9085-322fd166d13a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:10:40.936993 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:40.936885 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6eca4d-207a-4154-9085-322fd166d13a-kube-api-access-p2w25" (OuterVolumeSpecName: "kube-api-access-p2w25") pod "0d6eca4d-207a-4154-9085-322fd166d13a" (UID: "0d6eca4d-207a-4154-9085-322fd166d13a"). InnerVolumeSpecName "kube-api-access-p2w25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:10:41.035588 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:41.035568 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d6eca4d-207a-4154-9085-322fd166d13a-maas-api-tls\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:10:41.035588 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:41.035587 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d6eca4d-207a-4154-9085-322fd166d13a-openshift-service-ca\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:10:41.035719 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:41.035597 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2w25\" (UniqueName: \"kubernetes.io/projected/0d6eca4d-207a-4154-9085-322fd166d13a-kube-api-access-p2w25\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:10:41.642893 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:41.642865 2573 generic.go:356] "Generic (PLEG): container finished" podID="0d6eca4d-207a-4154-9085-322fd166d13a" containerID="5159016c9d1ec2779584ecfa282bcd77a88795d6f43a6fca2916ead961db0300" exitCode=0 Jul 05 20:10:41.643004 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:41.642907 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" Jul 05 20:10:41.643004 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:41.642947 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" event={"ID":"0d6eca4d-207a-4154-9085-322fd166d13a","Type":"ContainerDied","Data":"5159016c9d1ec2779584ecfa282bcd77a88795d6f43a6fca2916ead961db0300"} Jul 05 20:10:41.643004 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:41.642988 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2" event={"ID":"0d6eca4d-207a-4154-9085-322fd166d13a","Type":"ContainerDied","Data":"c20887c2c86447574c880beff7c050b9afbb2ea83b9fd13d2c497e6cf2ed8906"} Jul 05 20:10:41.643145 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:41.643014 2573 scope.go:117] "RemoveContainer" containerID="5159016c9d1ec2779584ecfa282bcd77a88795d6f43a6fca2916ead961db0300" Jul 05 20:10:41.653216 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:41.653200 2573 scope.go:117] "RemoveContainer" containerID="5159016c9d1ec2779584ecfa282bcd77a88795d6f43a6fca2916ead961db0300" Jul 05 20:10:41.653457 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:10:41.653437 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5159016c9d1ec2779584ecfa282bcd77a88795d6f43a6fca2916ead961db0300\": container with ID starting with 5159016c9d1ec2779584ecfa282bcd77a88795d6f43a6fca2916ead961db0300 not found: ID does not exist" containerID="5159016c9d1ec2779584ecfa282bcd77a88795d6f43a6fca2916ead961db0300" Jul 05 20:10:41.653520 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:41.653465 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5159016c9d1ec2779584ecfa282bcd77a88795d6f43a6fca2916ead961db0300"} err="failed to get container status \"5159016c9d1ec2779584ecfa282bcd77a88795d6f43a6fca2916ead961db0300\": rpc error: code = NotFound desc = could not find container \"5159016c9d1ec2779584ecfa282bcd77a88795d6f43a6fca2916ead961db0300\": container with ID starting with 5159016c9d1ec2779584ecfa282bcd77a88795d6f43a6fca2916ead961db0300 not found: ID does not exist" Jul 05 20:10:41.660211 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:41.660189 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2"] Jul 05 20:10:41.663810 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:41.663791 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-ec8ab7-75d756784b-qchr2"] Jul 05 20:10:43.469994 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:43.469962 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d6eca4d-207a-4154-9085-322fd166d13a" path="/var/lib/kubelet/pods/0d6eca4d-207a-4154-9085-322fd166d13a/volumes" Jul 05 20:10:45.277121 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:45.277087 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:10:52.844408 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:52.844369 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" podUID="a1928f92-9790-4a2e-ab50-59492eafe072" containerName="main" containerID="cri-o://62f437126c1de5dd66921565ba62cfb3546baaf2a7e51d011b762a275e5da024" gracePeriod=45 Jul 05 20:10:53.101201 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.101148 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:10:53.229979 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.229957 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-dshm\") pod \"a1928f92-9790-4a2e-ab50-59492eafe072\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " Jul 05 20:10:53.230118 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.230002 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dfgk\" (UniqueName: \"kubernetes.io/projected/a1928f92-9790-4a2e-ab50-59492eafe072-kube-api-access-7dfgk\") pod \"a1928f92-9790-4a2e-ab50-59492eafe072\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " Jul 05 20:10:53.230118 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.230032 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-tmp-dir\") pod \"a1928f92-9790-4a2e-ab50-59492eafe072\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " Jul 05 20:10:53.230118 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.230082 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-home\") pod \"a1928f92-9790-4a2e-ab50-59492eafe072\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " Jul 05 20:10:53.230118 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.230097 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-model-cache\") pod \"a1928f92-9790-4a2e-ab50-59492eafe072\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " Jul 05 20:10:53.230354 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.230127 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1928f92-9790-4a2e-ab50-59492eafe072-tls-certs\") pod \"a1928f92-9790-4a2e-ab50-59492eafe072\" (UID: \"a1928f92-9790-4a2e-ab50-59492eafe072\") " Jul 05 20:10:53.230399 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.230355 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-home" (OuterVolumeSpecName: "home") pod "a1928f92-9790-4a2e-ab50-59492eafe072" (UID: "a1928f92-9790-4a2e-ab50-59492eafe072"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:10:53.230399 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.230369 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a1928f92-9790-4a2e-ab50-59492eafe072" (UID: "a1928f92-9790-4a2e-ab50-59492eafe072"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:10:53.230399 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.230379 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-model-cache" (OuterVolumeSpecName: "model-cache") pod "a1928f92-9790-4a2e-ab50-59492eafe072" (UID: "a1928f92-9790-4a2e-ab50-59492eafe072"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:10:53.230569 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.230552 2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-tmp-dir\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:10:53.230619 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.230571 2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-home\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:10:53.230619 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.230584 2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-model-cache\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:10:53.232247 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.232216 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-dshm" (OuterVolumeSpecName: "dshm") pod "a1928f92-9790-4a2e-ab50-59492eafe072" (UID: "a1928f92-9790-4a2e-ab50-59492eafe072"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 20:10:53.232357 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.232256 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1928f92-9790-4a2e-ab50-59492eafe072-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a1928f92-9790-4a2e-ab50-59492eafe072" (UID: "a1928f92-9790-4a2e-ab50-59492eafe072"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:10:53.232357 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.232265 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1928f92-9790-4a2e-ab50-59492eafe072-kube-api-access-7dfgk" (OuterVolumeSpecName: "kube-api-access-7dfgk") pod "a1928f92-9790-4a2e-ab50-59492eafe072" (UID: "a1928f92-9790-4a2e-ab50-59492eafe072"). InnerVolumeSpecName "kube-api-access-7dfgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:10:53.330936 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.330914 2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1928f92-9790-4a2e-ab50-59492eafe072-dshm\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:10:53.330936 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.330933 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dfgk\" (UniqueName: \"kubernetes.io/projected/a1928f92-9790-4a2e-ab50-59492eafe072-kube-api-access-7dfgk\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:10:53.331077 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.330943 2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1928f92-9790-4a2e-ab50-59492eafe072-tls-certs\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:10:53.704531 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.704503 2573 generic.go:356] "Generic (PLEG): container finished" podID="a1928f92-9790-4a2e-ab50-59492eafe072" containerID="62f437126c1de5dd66921565ba62cfb3546baaf2a7e51d011b762a275e5da024" exitCode=0 Jul 05 20:10:53.704637 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.704542 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" Jul 05 20:10:53.704637 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.704604 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" event={"ID":"a1928f92-9790-4a2e-ab50-59492eafe072","Type":"ContainerDied","Data":"62f437126c1de5dd66921565ba62cfb3546baaf2a7e51d011b762a275e5da024"} Jul 05 20:10:53.704637 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.704636 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28" event={"ID":"a1928f92-9790-4a2e-ab50-59492eafe072","Type":"ContainerDied","Data":"dbe116dd694129875333e27757bece6341d4a2f9eecae01f51706c2e5b855ba7"} Jul 05 20:10:53.704752 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.704652 2573 scope.go:117] "RemoveContainer" containerID="62f437126c1de5dd66921565ba62cfb3546baaf2a7e51d011b762a275e5da024" Jul 05 20:10:53.713508 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.713489 2573 scope.go:117] "RemoveContainer" containerID="62f437126c1de5dd66921565ba62cfb3546baaf2a7e51d011b762a275e5da024" Jul 05 20:10:53.713781 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:10:53.713755 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f437126c1de5dd66921565ba62cfb3546baaf2a7e51d011b762a275e5da024\": container with ID starting with 62f437126c1de5dd66921565ba62cfb3546baaf2a7e51d011b762a275e5da024 not found: ID does not exist" containerID="62f437126c1de5dd66921565ba62cfb3546baaf2a7e51d011b762a275e5da024" Jul 05 20:10:53.713862 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.713790 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f437126c1de5dd66921565ba62cfb3546baaf2a7e51d011b762a275e5da024"} err="failed to get container status \"62f437126c1de5dd66921565ba62cfb3546baaf2a7e51d011b762a275e5da024\": rpc error: code = NotFound desc = could not find container \"62f437126c1de5dd66921565ba62cfb3546baaf2a7e51d011b762a275e5da024\": container with ID starting with 62f437126c1de5dd66921565ba62cfb3546baaf2a7e51d011b762a275e5da024 not found: ID does not exist" Jul 05 20:10:53.720653 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.720634 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28"] Jul 05 20:10:53.724489 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:53.724471 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-ec8ab7/rate-test-model-ec8ab7-kserve-bcbc99c6c-xhp28"] Jul 05 20:10:55.469162 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:10:55.469133 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1928f92-9790-4a2e-ab50-59492eafe072" path="/var/lib/kubelet/pods/a1928f92-9790-4a2e-ab50-59492eafe072/volumes" Jul 05 20:11:16.804256 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:16.804222 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:11:17.090110 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:17.090020 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:11:17.320737 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:17.320706 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:11:17.490808 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:17.490732 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:11:18.016308 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.016276 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:11:18.117960 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.117935 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:11:18.220770 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.220738 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45"] Jul 05 20:11:18.221010 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.220965 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" podUID="6f0bc2ee-2501-4225-85ba-33f400601f05" containerName="maas-api" containerID="cri-o://702254718b0c83dd29462a3cb1886ed2e982f298357d0871139849fe06e927ff" gracePeriod=30 Jul 05 20:11:18.246650 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.246624 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:11:18.665481 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.665462 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:11:18.821246 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.821219 2573 generic.go:356] "Generic (PLEG): container finished" podID="6f0bc2ee-2501-4225-85ba-33f400601f05" containerID="702254718b0c83dd29462a3cb1886ed2e982f298357d0871139849fe06e927ff" exitCode=0 Jul 05 20:11:18.821369 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.821258 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" Jul 05 20:11:18.821369 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.821272 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" event={"ID":"6f0bc2ee-2501-4225-85ba-33f400601f05","Type":"ContainerDied","Data":"702254718b0c83dd29462a3cb1886ed2e982f298357d0871139849fe06e927ff"} Jul 05 20:11:18.821369 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.821309 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45" event={"ID":"6f0bc2ee-2501-4225-85ba-33f400601f05","Type":"ContainerDied","Data":"b99b7181a3ebdeffee85095c2f988eaacaa15476bf176906d115a842bb7ac98c"} Jul 05 20:11:18.821369 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.821330 2573 scope.go:117] "RemoveContainer" containerID="702254718b0c83dd29462a3cb1886ed2e982f298357d0871139849fe06e927ff" Jul 05 20:11:18.825690 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.825669 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f0bc2ee-2501-4225-85ba-33f400601f05-maas-api-tls\") pod \"6f0bc2ee-2501-4225-85ba-33f400601f05\" (UID: \"6f0bc2ee-2501-4225-85ba-33f400601f05\") " Jul 05 20:11:18.825807 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.825743 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdt9g\" (UniqueName: \"kubernetes.io/projected/6f0bc2ee-2501-4225-85ba-33f400601f05-kube-api-access-pdt9g\") pod \"6f0bc2ee-2501-4225-85ba-33f400601f05\" (UID: \"6f0bc2ee-2501-4225-85ba-33f400601f05\") " Jul 05 20:11:18.825860 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.825819 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0bc2ee-2501-4225-85ba-33f400601f05-openshift-service-ca\") pod \"6f0bc2ee-2501-4225-85ba-33f400601f05\" (UID: \"6f0bc2ee-2501-4225-85ba-33f400601f05\") " Jul 05 20:11:18.826167 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.826145 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0bc2ee-2501-4225-85ba-33f400601f05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f0bc2ee-2501-4225-85ba-33f400601f05" (UID: "6f0bc2ee-2501-4225-85ba-33f400601f05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 20:11:18.827742 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.827720 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f0bc2ee-2501-4225-85ba-33f400601f05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f0bc2ee-2501-4225-85ba-33f400601f05" (UID: "6f0bc2ee-2501-4225-85ba-33f400601f05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 20:11:18.827827 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.827801 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0bc2ee-2501-4225-85ba-33f400601f05-kube-api-access-pdt9g" (OuterVolumeSpecName: "kube-api-access-pdt9g") pod "6f0bc2ee-2501-4225-85ba-33f400601f05" (UID: "6f0bc2ee-2501-4225-85ba-33f400601f05"). InnerVolumeSpecName "kube-api-access-pdt9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 20:11:18.829697 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.829667 2573 scope.go:117] "RemoveContainer" containerID="702254718b0c83dd29462a3cb1886ed2e982f298357d0871139849fe06e927ff" Jul 05 20:11:18.829909 ip-10-0-138-40 kubenswrapper[2573]: E0705 20:11:18.829892 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702254718b0c83dd29462a3cb1886ed2e982f298357d0871139849fe06e927ff\": container with ID starting with 702254718b0c83dd29462a3cb1886ed2e982f298357d0871139849fe06e927ff not found: ID does not exist" containerID="702254718b0c83dd29462a3cb1886ed2e982f298357d0871139849fe06e927ff" Jul 05 20:11:18.829961 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.829918 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702254718b0c83dd29462a3cb1886ed2e982f298357d0871139849fe06e927ff"} err="failed to get container status \"702254718b0c83dd29462a3cb1886ed2e982f298357d0871139849fe06e927ff\": rpc error: code = NotFound desc = could not find container \"702254718b0c83dd29462a3cb1886ed2e982f298357d0871139849fe06e927ff\": container with ID starting with 702254718b0c83dd29462a3cb1886ed2e982f298357d0871139849fe06e927ff not found: ID does not exist" Jul 05 20:11:18.926674 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.926655 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f0bc2ee-2501-4225-85ba-33f400601f05-openshift-service-ca\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:11:18.926674 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.926673 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f0bc2ee-2501-4225-85ba-33f400601f05-maas-api-tls\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:11:18.926783 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:18.926682 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdt9g\" (UniqueName: \"kubernetes.io/projected/6f0bc2ee-2501-4225-85ba-33f400601f05-kube-api-access-pdt9g\") on node \"ip-10-0-138-40.ec2.internal\" DevicePath \"\"" Jul 05 20:11:19.142533 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:19.142509 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45"] Jul 05 20:11:19.144471 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:19.144452 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-c11b28-68554cc7dd-kvs45"] Jul 05 20:11:19.469723 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:19.469667 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f0bc2ee-2501-4225-85ba-33f400601f05" path="/var/lib/kubelet/pods/6f0bc2ee-2501-4225-85ba-33f400601f05/volumes" Jul 05 20:11:23.672615 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:23.672578 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:11:23.772176 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:23.772149 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:11:23.944174 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:23.944103 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:11:24.161596 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:24.161567 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:11:24.577698 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:24.577674 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b5b7fd64f-vsx9f_0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2/authorino/0.log" Jul 05 20:11:29.283015 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:29.282987 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-6d694dc8c4-4kgb2_1c10c543-1810-4d6d-8b1d-e1afd66c88eb/maas-api/0.log" Jul 05 20:11:29.387731 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:29.387712 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29721345-74rpw_b4de75db-4fed-4039-ad76-93383a7c5052/cleanup/0.log" Jul 05 20:11:29.495101 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:29.495074 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29721360-n9kkh_190e4082-b68d-4a36-b36d-fae0208f214a/cleanup/0.log" Jul 05 20:11:29.610636 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:29.610580 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f64fbc48c-xbsjd_4ec720cb-1b72-4eff-b8f7-059e0be3ea12/manager/0.log" Jul 05 20:11:30.190448 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:30.190427 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68674bcb-mx6pq_62953dea-e74e-4fe8-a427-6a8a462f4287/copy-manifests/0.log" Jul 05 20:11:30.196091 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:30.196074 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68674bcb-mx6pq_62953dea-e74e-4fe8-a427-6a8a462f4287/manager/0.log" Jul 05 20:11:30.309855 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:30.309835 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-r78zp_5523224c-838f-4e26-b02f-802e67632aaa/postgres/0.log" Jul 05 20:11:31.569725 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:31.569703 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b5b7fd64f-vsx9f_0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2/authorino/0.log" Jul 05 20:11:33.065100 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:33.065071 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7484578f88-bj4ds_f6d3e615-ce5d-44a1-a62e-a43485f7c0ba/kube-auth-proxy/0.log" Jul 05 20:11:33.312440 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:33.312416 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-rkn5g_0258ded4-4884-4e4c-9124-8b9e227b222f/payload-pre-processing/0.log" Jul 05 20:11:33.533393 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:33.533372 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9d66c8c9c-mg7q2_9d9bd347-ceb3-4b36-8654-7f2ddbecf9af/router/0.log" Jul 05 20:11:33.871683 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:33.871633 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-x9jgd_2e03934f-dded-4ad0-99e7-92cc0f843a1d/main/0.log" Jul 05 20:11:34.092123 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:34.092101 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-92skn_15d38acb-947e-4c2d-b751-df98c72781b6/main/0.log" Jul 05 20:11:38.372008 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:38.371980 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:11:38.466448 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:38.466423 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" limit=1024 Jul 05 20:11:41.077224 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:41.077199 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pcsv4_d84a2165-5809-40c0-a5b3-7d3a976291ee/global-pull-secret-syncer/0.log" Jul 05 20:11:41.161617 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:41.161589 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ktlfv_dc4c3262-1cc8-4aea-a843-26a51eb356f8/konnectivity-agent/0.log" Jul 05 20:11:41.247520 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:41.247494 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-40.ec2.internal_c667c56910299e2ae0fff31b3d85acc6/haproxy/0.log" Jul 05 20:11:46.124364 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:46.124277 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b5b7fd64f-vsx9f_0da6f7e0-c5cb-402d-b4c4-c42cb443ecf2/authorino/0.log" Jul 05 20:11:48.266019 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.265993 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-xcnwf_90a56a02-5f5d-48bd-9285-c4db913af929/cluster-monitoring-operator/0.log" Jul 05 20:11:48.306287 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.306265 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-8mhrw_4219ed94-cc22-41e7-8d74-f9f4fbb199ee/kube-state-metrics/0.log" Jul 05 20:11:48.322776 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.322755 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-8mhrw_4219ed94-cc22-41e7-8d74-f9f4fbb199ee/kube-rbac-proxy-main/0.log" Jul 05 20:11:48.340957 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.340936 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-8mhrw_4219ed94-cc22-41e7-8d74-f9f4fbb199ee/kube-rbac-proxy-self/0.log" Jul 05 20:11:48.557378 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.557362 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b48b7_e44061fd-79c4-4cb4-b98f-0f4d848233c6/node-exporter/0.log" Jul 05 20:11:48.584486 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.584465 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b48b7_e44061fd-79c4-4cb4-b98f-0f4d848233c6/kube-rbac-proxy/0.log" Jul 05 20:11:48.605951 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.605921 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b48b7_e44061fd-79c4-4cb4-b98f-0f4d848233c6/init-textfile/0.log" Jul 05 20:11:48.649874 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.649861 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-xcjkp_ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c/kube-rbac-proxy-main/0.log" Jul 05 20:11:48.675994 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.675977 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-xcjkp_ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c/kube-rbac-proxy-self/0.log" Jul 05 20:11:48.696651 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.696634 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-xcjkp_ca3b51ff-cd85-43a1-b05b-72f32e2e3c4c/openshift-state-metrics/0.log" Jul 05 20:11:48.741593 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.741546 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b2e2692b-152b-4686-b5fc-13549d58457d/prometheus/0.log" Jul 05 20:11:48.760195 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.760176 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b2e2692b-152b-4686-b5fc-13549d58457d/config-reloader/0.log" Jul 05 20:11:48.780161 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.780146 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b2e2692b-152b-4686-b5fc-13549d58457d/thanos-sidecar/0.log" Jul 05 20:11:48.799402 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.799388 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b2e2692b-152b-4686-b5fc-13549d58457d/kube-rbac-proxy-web/0.log" Jul 05 20:11:48.818382 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.818362 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b2e2692b-152b-4686-b5fc-13549d58457d/kube-rbac-proxy/0.log" Jul 05 20:11:48.842178 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.842161 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b2e2692b-152b-4686-b5fc-13549d58457d/kube-rbac-proxy-thanos/0.log" Jul 05 20:11:48.862076 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:48.862055 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b2e2692b-152b-4686-b5fc-13549d58457d/init-config-reloader/0.log" Jul 05 20:11:49.041213 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:49.041192 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f9b4d8c9b-vq7vr_b910b1ae-d8e3-4201-a800-06319b8e02bf/thanos-query/0.log" Jul 05 20:11:49.062462 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:49.062445 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f9b4d8c9b-vq7vr_b910b1ae-d8e3-4201-a800-06319b8e02bf/kube-rbac-proxy-web/0.log" Jul 05 20:11:49.081235 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:49.081218 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f9b4d8c9b-vq7vr_b910b1ae-d8e3-4201-a800-06319b8e02bf/kube-rbac-proxy/0.log" Jul 05 20:11:49.100700 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:49.100683 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f9b4d8c9b-vq7vr_b910b1ae-d8e3-4201-a800-06319b8e02bf/prom-label-proxy/0.log" Jul 05 20:11:49.120286 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:49.120270 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f9b4d8c9b-vq7vr_b910b1ae-d8e3-4201-a800-06319b8e02bf/kube-rbac-proxy-rules/0.log" Jul 05 20:11:49.138962 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:49.138944 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6f9b4d8c9b-vq7vr_b910b1ae-d8e3-4201-a800-06319b8e02bf/kube-rbac-proxy-metrics/0.log" Jul 05 20:11:51.012555 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.012490 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/1.log" Jul 05 20:11:51.016450 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.016430 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-qb5w5_d50d6beb-5c77-4e3e-a170-a9220239ce86/console-operator/2.log" Jul 05 20:11:51.574273 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.574229 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-p6dfl_fa57f98a-83b8-4dfb-8c4e-97126b6b9cbe/download-server/0.log" Jul 05 20:11:51.691228 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.691197 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr"] Jul 05 20:11:51.691678 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.691660 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d6eca4d-207a-4154-9085-322fd166d13a" containerName="maas-api" Jul 05 20:11:51.691744 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.691682 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d6eca4d-207a-4154-9085-322fd166d13a" containerName="maas-api" Jul 05 20:11:51.691744 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.691694 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f0bc2ee-2501-4225-85ba-33f400601f05" containerName="maas-api" Jul 05 20:11:51.691744 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.691701 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f0bc2ee-2501-4225-85ba-33f400601f05" containerName="maas-api" Jul 05 20:11:51.691744 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.691722 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1928f92-9790-4a2e-ab50-59492eafe072" containerName="main" Jul 05 20:11:51.691744 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.691730 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1928f92-9790-4a2e-ab50-59492eafe072" containerName="main" Jul 05 20:11:51.691916 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.691799 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f0bc2ee-2501-4225-85ba-33f400601f05" containerName="maas-api" Jul 05 20:11:51.691916 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.691813 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d6eca4d-207a-4154-9085-322fd166d13a" containerName="maas-api" Jul 05 20:11:51.691916 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.691824 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1928f92-9790-4a2e-ab50-59492eafe072" containerName="main" Jul 05 20:11:51.695222 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.695204 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:51.697781 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.697763 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-27wzv\"/\"openshift-service-ca.crt\"" Jul 05 20:11:51.697910 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.697896 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-27wzv\"/\"kube-root-ca.crt\"" Jul 05 20:11:51.699222 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.699207 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-27wzv\"/\"default-dockercfg-62j8p\"" Jul 05 20:11:51.704501 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.704481 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr"] Jul 05 20:11:51.773704 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.773686 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1805ab53-7b03-4b10-96bb-2eaf3d8c55ed-lib-modules\") pod \"perf-node-gather-daemonset-wkzpr\" (UID: \"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:51.773807 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.773715 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1805ab53-7b03-4b10-96bb-2eaf3d8c55ed-proc\") pod \"perf-node-gather-daemonset-wkzpr\" (UID: \"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:51.773807 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.773737 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1805ab53-7b03-4b10-96bb-2eaf3d8c55ed-sys\") pod \"perf-node-gather-daemonset-wkzpr\" (UID: \"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:51.773807 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.773758 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8cm\" (UniqueName: \"kubernetes.io/projected/1805ab53-7b03-4b10-96bb-2eaf3d8c55ed-kube-api-access-rb8cm\") pod \"perf-node-gather-daemonset-wkzpr\" (UID: \"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:51.773906 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.773861 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1805ab53-7b03-4b10-96bb-2eaf3d8c55ed-podres\") pod \"perf-node-gather-daemonset-wkzpr\" (UID: \"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:51.874702 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.874651 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1805ab53-7b03-4b10-96bb-2eaf3d8c55ed-lib-modules\") pod \"perf-node-gather-daemonset-wkzpr\" (UID: \"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:51.874702 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.874687 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1805ab53-7b03-4b10-96bb-2eaf3d8c55ed-proc\") pod \"perf-node-gather-daemonset-wkzpr\" (UID: \"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:51.874819 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.874718 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1805ab53-7b03-4b10-96bb-2eaf3d8c55ed-sys\") pod \"perf-node-gather-daemonset-wkzpr\" (UID: \"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:51.874819 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.874785 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1805ab53-7b03-4b10-96bb-2eaf3d8c55ed-sys\") pod \"perf-node-gather-daemonset-wkzpr\" (UID: \"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:51.874819 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.874792 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1805ab53-7b03-4b10-96bb-2eaf3d8c55ed-proc\") pod \"perf-node-gather-daemonset-wkzpr\" (UID: \"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:51.874932 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.874817 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1805ab53-7b03-4b10-96bb-2eaf3d8c55ed-lib-modules\") pod \"perf-node-gather-daemonset-wkzpr\" (UID: \"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:51.874932 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.874817 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8cm\" (UniqueName: \"kubernetes.io/projected/1805ab53-7b03-4b10-96bb-2eaf3d8c55ed-kube-api-access-rb8cm\") pod \"perf-node-gather-daemonset-wkzpr\" (UID: \"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:51.874932 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.874912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1805ab53-7b03-4b10-96bb-2eaf3d8c55ed-podres\") pod \"perf-node-gather-daemonset-wkzpr\" (UID: \"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:51.875033 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.875018 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1805ab53-7b03-4b10-96bb-2eaf3d8c55ed-podres\") pod \"perf-node-gather-daemonset-wkzpr\" (UID: \"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:51.881638 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:51.881619 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8cm\" (UniqueName: \"kubernetes.io/projected/1805ab53-7b03-4b10-96bb-2eaf3d8c55ed-kube-api-access-rb8cm\") pod \"perf-node-gather-daemonset-wkzpr\" (UID: \"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed\") " pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:52.005678 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:52.005654 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:52.130310 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:52.130287 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr"] Jul 05 20:11:52.132387 ip-10-0-138-40 kubenswrapper[2573]: W0705 20:11:52.132358 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1805ab53_7b03_4b10_96bb_2eaf3d8c55ed.slice/crio-7d5f7520cdb9dcfaaa98024e771dfefec7d87fae8cea3664a19adee11a4917d2 WatchSource:0}: Error finding container 7d5f7520cdb9dcfaaa98024e771dfefec7d87fae8cea3664a19adee11a4917d2: Status 404 returned error can't find the container with id 7d5f7520cdb9dcfaaa98024e771dfefec7d87fae8cea3664a19adee11a4917d2 Jul 05 20:11:52.309574 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:52.309552 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-9rf6c_619c0eb1-3880-4758-b423-9acf269979c0/volume-data-source-validator/0.log" Jul 05 20:11:52.981371 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:52.981338 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" event={"ID":"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed","Type":"ContainerStarted","Data":"beebf2d6508884c285692e647ba101d1ef5c397cf74b676a43eed6be003bcdc2"} Jul 05 20:11:52.981514 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:52.981398 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:52.981514 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:52.981420 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" event={"ID":"1805ab53-7b03-4b10-96bb-2eaf3d8c55ed","Type":"ContainerStarted","Data":"7d5f7520cdb9dcfaaa98024e771dfefec7d87fae8cea3664a19adee11a4917d2"} Jul 05 20:11:52.998121 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:52.998075 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" podStartSLOduration=1.9980616009999999 podStartE2EDuration="1.998061601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 20:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 20:11:52.995086456 +0000 UTC m=+2513.995104560" watchObservedRunningTime="2026-07-05 20:11:52.998061601 +0000 UTC m=+2513.998079701" Jul 05 20:11:53.295802 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:53.295776 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6vhql_9d64b827-c0f3-4a50-8e75-6284cffcf625/dns/0.log" Jul 05 20:11:53.316499 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:53.316481 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6vhql_9d64b827-c0f3-4a50-8e75-6284cffcf625/kube-rbac-proxy/0.log" Jul 05 20:11:53.470623 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:53.470603 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xnmxp_5c96e505-de6a-4853-822c-e80de519e002/dns-node-resolver/0.log" Jul 05 20:11:53.978358 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:53.978336 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4snds_cb2e6493-927f-48b2-8e47-16dc2379911a/node-ca/0.log" Jul 05 20:11:55.120934 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:55.120908 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7484578f88-bj4ds_f6d3e615-ce5d-44a1-a62e-a43485f7c0ba/kube-auth-proxy/0.log" Jul 05 20:11:55.228157 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:55.228140 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-rkn5g_0258ded4-4884-4e4c-9124-8b9e227b222f/payload-pre-processing/0.log" Jul 05 20:11:55.303331 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:55.303310 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9d66c8c9c-mg7q2_9d9bd347-ceb3-4b36-8654-7f2ddbecf9af/router/0.log" Jul 05 20:11:55.930606 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:55.930587 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4vprg_85f8e787-c095-491d-91e6-2840a6364447/serve-healthcheck-canary/0.log" Jul 05 20:11:56.406255 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:56.406228 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-wz8gp_bc38af11-062b-458d-8f23-4d34f9fd1fdd/insights-operator/0.log" Jul 05 20:11:56.406604 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:56.406259 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-wz8gp_bc38af11-062b-458d-8f23-4d34f9fd1fdd/insights-operator/1.log" Jul 05 20:11:56.487966 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:56.487948 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8f8zn_68db1a0c-9074-4478-83cf-e9a5b517b553/kube-rbac-proxy/0.log" Jul 05 20:11:56.506292 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:56.506275 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8f8zn_68db1a0c-9074-4478-83cf-e9a5b517b553/exporter/0.log" Jul 05 20:11:56.525203 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:56.525169 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8f8zn_68db1a0c-9074-4478-83cf-e9a5b517b553/extractor/0.log" Jul 05 20:11:58.847521 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:58.847495 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-6d694dc8c4-4kgb2_1c10c543-1810-4d6d-8b1d-e1afd66c88eb/maas-api/0.log" Jul 05 20:11:58.865961 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:58.865937 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29721345-74rpw_b4de75db-4fed-4039-ad76-93383a7c5052/cleanup/0.log" Jul 05 20:11:58.887353 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:58.887323 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29721360-n9kkh_190e4082-b68d-4a36-b36d-fae0208f214a/cleanup/0.log" Jul 05 20:11:58.978234 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:58.978216 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f64fbc48c-xbsjd_4ec720cb-1b72-4eff-b8f7-059e0be3ea12/manager/0.log" Jul 05 20:11:58.995517 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:58.995496 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-27wzv/perf-node-gather-daemonset-wkzpr" Jul 05 20:11:59.245052 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:59.244975 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68674bcb-mx6pq_62953dea-e74e-4fe8-a427-6a8a462f4287/manager/0.log" Jul 05 20:11:59.262566 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:59.262551 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68674bcb-mx6pq_62953dea-e74e-4fe8-a427-6a8a462f4287/copy-manifests/0.log" Jul 05 20:11:59.283144 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:11:59.283124 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-r78zp_5523224c-838f-4e26-b02f-802e67632aaa/postgres/0.log" Jul 05 20:12:00.623830 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:00.623810 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-db9cc5d86-fgqmv_05a48ac2-0f73-4290-b6c2-f3e0a7262452/manager/0.log" Jul 05 20:12:02.077573 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:02.077506 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7484578f88-bj4ds_f6d3e615-ce5d-44a1-a62e-a43485f7c0ba/kube-auth-proxy/0.log" Jul 05 20:12:02.175626 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:02.175597 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-rkn5g_0258ded4-4884-4e4c-9124-8b9e227b222f/payload-pre-processing/0.log" Jul 05 20:12:02.242702 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:02.242683 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9d66c8c9c-mg7q2_9d9bd347-ceb3-4b36-8654-7f2ddbecf9af/router/0.log" Jul 05 20:12:03.627273 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:03.627247 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7484578f88-bj4ds_f6d3e615-ce5d-44a1-a62e-a43485f7c0ba/kube-auth-proxy/0.log" Jul 05 20:12:03.720853 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:03.720826 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-rkn5g_0258ded4-4884-4e4c-9124-8b9e227b222f/payload-pre-processing/0.log" Jul 05 20:12:03.784827 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:03.784810 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-9d66c8c9c-mg7q2_9d9bd347-ceb3-4b36-8654-7f2ddbecf9af/router/0.log" Jul 05 20:12:05.984146 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:05.984121 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-k8pzp_81eba20f-cd1e-41cb-a868-ee8f133a20ef/kube-storage-version-migrator-operator/1.log" Jul 05 20:12:05.985016 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:05.984995 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-k8pzp_81eba20f-cd1e-41cb-a868-ee8f133a20ef/kube-storage-version-migrator-operator/0.log" Jul 05 20:12:07.200177 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:07.200103 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v428c_2b2a331e-da1e-4db5-ac53-84d90da88ff3/kube-multus-additional-cni-plugins/0.log" Jul 05 20:12:07.219590 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:07.219569 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v428c_2b2a331e-da1e-4db5-ac53-84d90da88ff3/egress-router-binary-copy/0.log" Jul 05 20:12:07.240084 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:07.240036 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v428c_2b2a331e-da1e-4db5-ac53-84d90da88ff3/cni-plugins/0.log" Jul 05 20:12:07.258351 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:07.258329 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v428c_2b2a331e-da1e-4db5-ac53-84d90da88ff3/bond-cni-plugin/0.log" Jul 05 20:12:07.276600 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:07.276569 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v428c_2b2a331e-da1e-4db5-ac53-84d90da88ff3/routeoverride-cni/0.log" Jul 05 20:12:07.294815 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:07.294796 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v428c_2b2a331e-da1e-4db5-ac53-84d90da88ff3/whereabouts-cni-bincopy/0.log" Jul 05 20:12:07.313186 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:07.313167 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-v428c_2b2a331e-da1e-4db5-ac53-84d90da88ff3/whereabouts-cni/0.log" Jul 05 20:12:07.344664 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:07.344648 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l7gg4_cfaa8adb-2377-43ab-8481-f3d30e66a994/kube-multus/0.log" Jul 05 20:12:07.394186 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:07.394160 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-86bxp_ed74973b-94ab-43a3-a0d3-b5e89daf5cf3/network-metrics-daemon/0.log" Jul 05 20:12:07.411209 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:07.411190 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-86bxp_ed74973b-94ab-43a3-a0d3-b5e89daf5cf3/kube-rbac-proxy/0.log" Jul 05 20:12:08.431652 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:08.431627 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4jh7_46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c/ovn-controller/0.log" Jul 05 20:12:08.446739 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:08.446716 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4jh7_46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c/ovn-acl-logging/0.log" Jul 05 20:12:08.466151 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:08.466130 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4jh7_46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c/kube-rbac-proxy-node/0.log" Jul 05 20:12:08.483704 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:08.483680 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4jh7_46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c/kube-rbac-proxy-ovn-metrics/0.log" Jul 05 20:12:08.500424 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:08.500404 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4jh7_46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c/northd/0.log" Jul 05 20:12:08.518580 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:08.518565 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4jh7_46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c/nbdb/0.log" Jul 05 20:12:08.536559 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:08.536543 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4jh7_46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c/sbdb/0.log" Jul 05 20:12:08.594554 ip-10-0-138-40 kubenswrapper[2573]: I0705 20:12:08.594536 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-s4jh7_46dae9aa-7ed9-4e1c-bb2c-f25dfd1b114c/ovnkube-controller/0.log"